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Introduction 


Family  support  programs  have  been  implemented  across  a 
wide  variety  of  service  systems  for  children  and  families 
(Ayoub,  Daro,  Landsverk,  & Lutzker,  1996).  Development 
of  effective  knowledge  about  "what  works"  in  these 
programs  has  been  hampered  by  a number  of  program 
implementation  and  evaluation  issues  (Pecora,  Fraser, 
Nelson,  McCroskey,  & Meezan,  1995).  These  include  lack 
of  a coherent  theory  of  family-based  services,  poorly 
specified  program  models,  inconsistent  referral  and  case 
targeting  practices,  absence  of  procedural  guidelines  for 
direct  service  issues  (e.g.,  staff  selection,  training,  and 
retention),  and  an  inability  to  develop  a systematic 
mechanism  for  monitoring  program  quality  and  refining  the 
program  as  it  stabilizes  and  matures. 

In  this  summary,  a local  family  support  program  is 
presented  in  order  to  illustrate  these  issues  and  highlight  the 
importance  of  appropriate  planning,  monitoring,  and 
evaluation  of  family  support  efforts.  The  Pecora  et  al. 

(1995)  framework  is  also  expanded  by  describing  the 
significance  of  contextual  factors  which  affect  program 
implementation  and  evaluation;  such  a description  has 
implications  for  third-party  funders  of  services,  providers, 
and  evaluators. 
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Method 


Total  Family  Strategy  (TFS)  is  a home-based  family  support 
program  funded  by  the  Juvenile  Welfare  Board  (JWB)  of 
Pinellas  County,  Florida.  JWB  is  an  independent  special 
taxing  district  that  funds  delivery  of  children's  services 
through  other  community  agencies  (Juvenile  Welfare  Board, 
1992). 

This  program  was  initiated  by  JWB  as  a means  of 
introducing  a family  empowerment  model  of  early 
intervention  into  the  array  of  existing  services  for  children 
and  families.  The  goal  of  TFS  is  to  prevent  "million-dollar 
families"  (Friedman,  1991),  who  could  incur  that  amount  in 
service  system  costs  throughout  the  life  of  the  family.  The 
target  population  is  young  families  with  children  under  the 
age  of  12  who  are  at  risk  of  developing  serious  problems, 
and  who  have  not  already  utilized  significant  social  service 
resources.  Four  agencies  in  the  community  provide  this 
service.  Program  components  include  case  management, 
access  to  community  resources,  advocacy,  and  counseling. 
The  University  of  South  Florida,  Florida  Mental  Health 
Institute  (FMHI)  has  been  involved  from  the  inception  of 
the  program  in  conceptualizing  the  program  and  evaluating 
its  impact. 

Beginning  in  1994,  a process  evaluation  of  the  TFS  program 
was  conducted  (Silver  & Marfo,  1994).  Multiple  methods 
were  used  in  the  TFS  evaluation,  including  (a)  review  of 
existing  program  descriptions  and  other  documents,  (b) 
interviews  with  agency  directors  and  staff,  and  (c) 
quantitative  analysis  of  data  routinely  collected  from 
families  (N  = 93).  Intensive  case  studies  were  also 
conducted  with  TFS  families  (N  = 8)  and  staff. 

Results 

Process  evaluation  revealed  a number  of  issues  identified  by 
Pecora  et  al.  (1995)  operating  in  the  TFS  program,  as 
follows. 

Under-specification  of  the  program  model.  Interviews 
conducted  with  agency  directors  revealed  different  ideas 
about  the  purpose  of  the  program,  its  target  population,  and 
indicators  of  program  effectiveness.  For  example,  responses 
to  questions  about  the  purpose  of  the  program  revealed  a 
range  of  program  philosophies,  from  a focus  on  preventing 
serious  problems,  such  as  out-of-home  placements  or 
juvenile  delinquency,  to  an  emphasis  on  empowering 
families  to  problem-solve,  access  resources,  develop 
informal  networks,  and  become  more  self-sufficient.  These 
differences  in  program  philosophy  also  occurred  at  the 
program  staff  level,  as  revealed  by  case  study  interviews 
and  observations  at  monthly  staff  meetings. 

Inappropriate  referral  and  case  targeting.  Case  studies  and 
quantitative  analysis  indicated  that  many  families  being 
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quantitative  analysis  indicated  that  many  families  being 
served  in  the  program  had  already  experienced  significant 
systems  involvement.  Examples  from  the  case  studies 
included  involvement  in  the  child  protective  services 
system,  adult  corrections,  and  severe  emotional  difficulties. 
This  population  differed  significantly  from  the  one  that  the 
program  was  designed  to  serve;  moreover,  the  population 
which  was  served  varied  by  agency  (Silver,  1995).  Historic 
differences  in  populations  served,  referral  sources,  and 
funding  sources  may  have  contributed  to  different 
interpretations  of  the  eligibility  guidelines. 

Program  drift.  The  four  agencies  appeared  to  have  a similar 
picture  of  the  program  model  when  the  initiative  began,  as 
evidenced  by  a set  of  guidelines  they  had  jointly  developed. 
However,  agencies  appeared  to  interpret  the  guidelines 
differently,  which  led  to  differences  in  program 
implementation.  For  example,  as  a result  of  differences  in 
agencies'  staffing  patterns,  differences  occurred  in  the 
degree  to  which  therapeutic  services,  versus  case 
management  and  referral,  were  emphasized  in  staffs  work 
with  families.  Record-keeping  procedures  were  modified  to 
meet  requirements  of  the  larger  agency,  which  sometimes 
affected  methods  used  in  family  assessment  and  case  plan 
development.  Staff  turn-over,  combined  with  lack  of 
detailed  implementation  procedures,  increased  the 
opportunity  for  differences  in  the  interpretation  of  the 
program  model. 

Discussion 

Process  evaluation  identified  significant  variation  in 
agencies'  implementation  of  this  family  support  program. 

As  a result  of  this  and  other  information,  program  staff, 
JWB,  and  the  evaluator  are  developing  mechanisms  to 
improve  implementation  on  the  issues  described  above. 

There  were  important  contextual  issues  in  the  current 
initiative  that  shaped  program  implementation,  monitoring, 
and  evaluation  activities.  The  first  issue  pertains  to  the 
relationship  between  funders  and  service  providers.  In  1991, 
JWB  and  FMHI  identified  the  need  for  an  innovative 
service  and  sought  to  have  this  service  provided  by  local 
agencies.  The  impetus  for  the  initiative  came  from  outside 
the  agencies.  A second,  related  contextual  issue  is  that  the 
model  underlying  TFS  represented  a change  in  agencies' 
existing  practices.  Total  Family  Strategy  emphasizes  a 
home-based  (versus  office-based)  model,  working  with 
families  holistically  instead  of  with  the  identified  client. 
Additionally,  TFS  focused  on  partnering  with  families, 
building  on  their  strengths  and  providing  community-level 
supports,  as  opposed  to  employing  the  traditional 
therapeutic  approach.  Despite  the  joint  development  of 
program  guidelines,  the  process  evaluation  indicated  that 
agencies  were  operating  with  different  interpretations  of 
how  this  model  translated  into  practice.  This 
underspecification  of  the  program  model  led  to  uneven 
program  implementation  across  agencies. 
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Contextual  factors  also  affected  quality  control  and  program 
evaluation  activities.  Initially,  neither  JWB  nor  FMHI  folly 
realized  the  complexity  of  this  initiative,  including  the  time 
needed  to  implement  and  bring  about  the  desired  changes. 
As  the  program  developed,  recognition  of  the  necessity  for 
developing  new  approaches  to  contract  management  and 
evaluation  also  occurred.  "Lessons  learned"  included 
changing  from  a practice  of  separate  JWB  contract 
managers  (i.e.,  one  for  each  agency)  to  one  contract 
manager  who  worked  across  all  four  agencies  and  assumed 
a greater  coordinating  role. 

Pecora  et  al.  (1995)  note  that  one  of  the  pitfalls  of 
evaluating  programs  is  a mismatch  between  stage  of 
program  development  and  evaluation  design.  The  initial 
evaluation  design  focused  on  short-term  outcomes  (i.e.,  an 
approach  consistent  with  JWB's  practice  of  monitoring 
contracts  with  established  programs).  As  the  complexity  of 
managing  this  program  revealed  itself,  JWB  requested  an 
evaluation  plan  that  included  both  process  and  outcome 
components.  Process  evaluation  revealed  a number  of  issues 
that  will  impact  long-term  evaluation  of  this  program  (e.g., 
the  differences  in  the  populations  served  by  agencies  will 
require  analysis  of  families'  long-term  progress  separately 
for  each  agency).  This  has  led  to  a clearer  understanding  of 
the  limitations  of  the  evaluation  that  would  not  have  been 
achieved  without  a focus  on  process  evaluation. 

The  nature  of  family-support  programs  affects  the  context, 
implementation,  and  evaluation  of  programs.  Family 
support  services  offer  a wide  breadth  of  services,  and  issues 
faced  by  families  represent  multiple  domains  (e.g.,  housing, 
basic  needs,  transportation,  additional  education/training, 
parenting  support,  and  counseling).  The  heterogeneity  of  the 
target  population  poses  significant  challenges  for  family 
support  programs.  In  the  current  program,  these  challenges 
may  have  been  compounded  by  the  difficulty  of 
operationalizing  characteristics  of  those  likely  to  become 
"million  dollar  families."  In  addition  to  significant  process 
evaluation  issues,  family  support  programs  present 
formidable  challenges  to  outcome  evaluation.  Evaluation  is 
made  more  difficult  by  the  use  of  individualized  criteria  for 
determining  family  progress  and  success,  and  by  the  fact 
that  family  success  is  also  dependent  upon  available  and 
accessible  community  resources&endash;issues  which  are 
often  not  under  the  control  of  program  staff  (Silver,  1995). 

Experience  with  the  TFS  evaluation  leads  to  several 
recommendations.  These  recommendations  apply  to 
funders,  providers,  and  evaluators  of  family-based 
programs.  The  importance  of  careful,  systematic 
implementation  planning  cannot  be  overlooked.  This  starts 
with  review  of  Pecora,  et  al.'s  (1995)  description  of  the 
issues  involved  in  implementing  and  evaluating  services.  It 
is  also  crucial  to  allocate  sufficient  time  and  resources  to 
monitor  program  implementation.  The  specific  strategy  used 
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to  monitor  implementation  can  take  a number  of  forms. 
These  include  having  the  funder  and  providers  develop 
self-assessment  mechanisms  that  would  include 
"benchmarks  of  progress"  to  determine  whether  program 
implementation  is  on  track  (e.g.,  Is  there  a mechanism  for 
regular  review  and  refinements  of  the  program  model?  What 
information  is  being  collected  to  provide  feedback  about 
these  issues?).  If  the  time  needed  to  develop,  implement, 
and  monitor  the  program  exceeds  that  of  the  funder  and/or 
providers,  consider  having  an  external  evaluator  conduct  a 
process  evaluation.  A third  strategy  combines  elements  of 
the  first  two  (e.g.,  having  an  evaluator  provide  technical 
assistance  on  developing  self-assessment  measures). 

Carefully  consider  the  context  in  which  this  new  program  is 
being  implemented.  Is  this  an  innovative  service  that 
challenges  traditional  practices?  Who  is  in  charge  of  the 
innovation?  How  does  this  model  fit  with  existing 
philosophy,  practice,  and  training  of  providers?  What  group 
processes  can  be  used  to  facilitate  the  empowerment  of 
providers  and  their  ownership  of  innovation?  Be  sure  to 
define  roles,  including  the  role  of  the  evaluator.  Consider 
how  the  role  of  an  evaluator  can  be  utilized  to  help  develop 
a more  participatory  approach  to  program  evaluation  (e.g., 
as  a participant  observer  who  provides  technical  assistance 
and  helps  facilitate  the  group  process,  vs.  a traditional 
evaluator  that  may  lead  providers  to  comply,  rather  than 
take  risks  and  try  an  innovation). 

The  process  issues  described  in  this  summary  raise 
questions  about  how  providers  and  funders  can  work 
together  to  develop  and  deliver  innovative  family  support 
programs.  Developing  mechanisms  to  foster  collaboration 
may  be  especially  important  when  the  new  initiative 
requires  changes  in  existing  funder-provider  relationships 
and  staff  practices.  These  issues  are  likely  to  grow  in 
importance  as  family  support  programs  grow  and  pressures 
for  greater  accountability  of  programs  increase. 
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Introduction 

Intensive  family  preservation  services  (IFPS)  have  emerged 
as  a service  innovation  in  both  the  child  welfare  and 
children's  mental  health  service  systems.  IFPS  interventions 
involve  the  provision  of  intensive,  home-based, 
skill-focused  services  to  families  whose  children  are  at 
I imminent  risk  of  placement.  Colorado  has  implemented 
IFPS  as  part  of  its  system  of  care  for  children  with 
emotional  disabilities  and  their  families.  Existing  research 
on  IFPS  focuses  on  the  child  welfare  system,  with  primary 
attention  given  to  the  outcome  of  placement  prevention. 
Increasingly,  the  focus  is  shifting  toward  direct  measures  of 
client  functioning  and  its  relationship  to  placement 
prevention.  Several  studies  (Nelson,  1988;  Bath,  Richey,  & 
Haapala,  1992;  Fraser,  Pecora,  & Lewis,  1991)  indicate  that 
problems  in  children's  functioning  are  correlates  of 
placement. 

For  the  children's  mental  health  system,  child  level  of 
functioning  is  an  increasingly  important  variable,  with 
interest  driven  by  (a)  theoretical  shifts  from  psychodynamic 
and  behavioral  models  toward  ecological  models  for 
intervention  and  research,  and  (b)  measurement  issues, 
moving  from  diagnostic  categories  to  Level  of  Functioning 
(LOF)  measures.  One  such  measure  is  the  Colorado  Client 
Assessment  Record  (CCAR)  which  assesses  nine, 
domain-specific  areas  of  functioning  (Ellis,  Wackwitz,  & 
Foster,  1991). 

This  summary  focuses  on  three  areas  of  investigation:  (a) 
description  of  the  mental  health  family  preservation 
intervention,  (b)  validity  of  the  CCAR  as  a measurement  of 
children's  level  of  functioning,  and  (c)  identification  of  the 
multivariate  predictors  of  placement. 
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Method 


Service  Context  of  IFPS  in  Colorado's  Mental  Health 
System 

Establishing  a complete  description  of  service  innovations  is 
imperative  prior  to  examination  of  client  outcomes  and 
correlates  of  those  outcomes.  In  addition,  existing  research 
in  family  preservation  indicates  that  heterogeneity  of  service 
evolution  is  common  across  sites  (Schuerman,  Rzepnicki, 
Littell,  & Chak,  1993;  McDonald  & Associates,  1990).  Site 
visits  were  conducted  at  each  of  the  eight  IFPS  service  sites, 
including  interviews  with  26  program  directors,  supervisory, 
and  service  staff.  Qualitative  interview  data  was  gathered  in 
the  form  of  taped  interviews,  each  of  which  followed  a 
semi-structured  interview  guide  focusing  on  service 
philosophy,  intervention  models,  and  agency-community 
contexts. 

The  differing  environments  in  which  programs  operate  have 
driven  slight  divergence  in  program  models,  with  the 
primary  impacts  on  intervention  length  and  mechanisms  for 
program  access.  These  differences  are  small,  however,  and 
must  be  considered  in  the  context  of  the  many  similarities 
among  service  sites.  Programs  evidenced  striking 
similarities  in  philosophies  of  service  and  types  of 
interventions  offered.  The  dominant  service  themes  were: 

(a)  the  power  and  nature  of  the  helping  relationship,  (b)  the 
strength  of  the  crisis  intervention/brief  intervention 
foundation,  and  (c)  the  commitment  to  skill-focused 
intervention. 

Factorial  Structure  of  the  Colorado  Client  Assessment 
Record 

Level  of  functioning  measures  can  provide  information 
important  to  tailoring  interventions,  predicting  success  of 
these  interventions,  and  describing  outcomes.  The  CCAR  is 
administered  for  all  Department  of  Mental  Health  (DMH) 
clients,  including  children;  however,  no  exploration  of  its 
factorial  structure  had  been  done  for  a child  mental  health 
population.  The  CCAR  assesses  mental  health  functioning 
in  nine  domains  (Feeling/Mood/ Affect,  Thinking, 
Medical/Physical,  Substance  Use,  Family,  Interpersonal, 
Role  Performance,  Social  Legal,  and  Self  care/basic  needs) 
using  a combination  of  rating  scales  and  checklist  items.  In 
order  to  explore  the  utility  of  this  measure,  it  was 
hypothesized  that  the  nine  factor  model  which  forms  the 
conceptual  organization  of  the  instrument  best  fits  the  data 
obtained  from  DMH  clients. 

Each  of  the  nine  Level  of  Functioning  (LOF)  scales  is 
measured  on  a 0 to  50  point  scale.  Of  78  problem  checklist 
items,  69  are  conceptually  linked  to  the  specific  level  of 
functioning  scales  and  describe  attributes  of  the  client  and 
the  client's  interaction  with  their  environment.  Fourteen 
Personal  Problem  Profile  (PPP)  scales  were  developed  from 
these  checklist  items. 


these  checklist  items. 

Subjects 

The  sample  consisted  of  a randomly  selected  group  of  5,000 
child  recipients  of  mental  health  services.  Data  from  a 50% 
sample  was  used  for  exploratory  factor  analysis:  scores  from 
the  remainder  were  used  for  confirmatory  factor  analysis, 
using  the  EQS  structural  equation  modeling  program. 

Results 

Analysis 

The  9 Level  of  Functioning  scales  and  14  Personal  Problem 
Profile  scales  were  submitted  to  exploratory  factor  analysis 
using  maximum  likelihood  factoring  and  oblimin  rotation 
due  to  the  theoretical  likelihood  that  dimensions  of 
functioning  are  inter-related.  Using  the  1.0  cutoff  for 
evaluation  of  eigenvalues,  a seven  factor  solution  emerged 
explaining  49.7%  of  variance,  with  all  LOF  scales  factoring 
with  their  associated  PPP  scales.  Two  sets  of  LOFs  and 
PPPs  factor  together:  (a)  Thinking  and  Self  care/Basic 
needs,  and  (b)  Role  performance  and  Socio-legal. 
Correlations  between  factors  range  from  .04  to  .43. 

Because  of  the  theoretical  interest  in  a nine  factor  model, 
this  solution  was  also  examined  (see  Figure  IV  This  model 
explains  56.3  % of  the  variance.  All  PPP  indicators  load 
strongly  with  their  associated  LOF  scales  on  a separate 
factor.  No  indicators  cross  load  to  other  factors.  Correlations 
between  the  factors  range  from  .05  to  .46.  Both  models  were 
carried  forward  into  the  confirmatory  factor  analysis  (CFA) 
using  EQS  (Bentler,  1989). 

The  seven  factor  model  did  not  fit  the  data,  with  a CFI  of 
.796.  Examination  of  the  modification  indices  indicated  that 
upwards  of  20  cross  loadings  of  indicators  were  necessary  to 
achieve  adequate  fit.  The  nine-factor  model  was  also  tested. 
Maximum  likelihood  factoring  was  used,  and  no  special 
problems  were  observed  during  optimization.  This  model 
exhibited  improved  fit,  with  a CFI  of  .867,  and  examination 
of  the  modification  indices  indicated  that  relatively  few 
cross  loadings  were  necessary  to  achieve  fit.  Allowing  five 
of  23  indicators  to  cross  load  produced  a model  which  fit 
the  data  with  a CFI  of  .900.  Three  of  these  cross  loadings, 
however,  were  very  small  (i.e.,  less  than  .2).  The 
standardized  structural  matrix  is  presented  in  Table  I. 

It  is  important  to  consider  possible  reasons  for  inadequate  fit 
of  the  hypothesized  model.  The  model  may  simply  not  fit 
these  data  because  it  does  not  contain  important  theoretical 
elements.  However,  there  may  also  be  technical  reasons. 
Monte  Carlo  studies  indicate  that  "large"  models,  even 
when  known  to  fit  perfectly,  generate  fit  estimates  lower 
than  "small"  models.  In  addition,  while  large  sample  sizes 
are  needed  to  achieve  stability  in  matrices,  very  large 
sample  sizes  impose  power  constraints  on  model  fitting,  as 
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sample  sizes  impose  power  constraints  on  model  fitting,  as 
do  high  factor  loadings  (Gerbing  & Anderson,  1993).  In  this 
model,  factor  loadings  are  high,  and  the  sample  and  model 
sizes  are  large. 

Given  the  evolving  understanding  of  model  fit  in  the  field  of 
structural  equation  modeling,  it  would  be  premature  to 
reject  the  hypothesized  9 factor  model  based  on  the  .90 
cutoff  for  the  CFI.  These  analyses  do  support  the  use  of  the 
existing  Level  of  Functioning  scales,  which  were  carried 
forward  into  subsequent  multivariate  analysis. 

Multivariate  Predictors  of  Placement  Outcomes 

While  level  of  functioning  data  are  instructive,  LOF  is  only 
one  variable  among  many  that  contribute  to  the  impact  of  an 
intervention  for  individual  families.  To  identify  which 
variables  had  the  most  power  to  predict  placement  outcomes 
for  the  family  preservation  intervention,  we  looked  at  a 
family  preservation  sample  that  consisted  of  the  316 
children  who  received  family  preservation  services  between 
1990  and  1992.  Data  collected  included  information  on 
demographics,  child  level  of  functioning  (CCAR),  service 
needs,  interventions,  and  placement  outcome.  Logistic 
regression  models,  using  backward  selection  with  a 
likelihood  ratio  criteria,  were  developed  to  examine  the 
correlates  of  placement  prevention  and  to  construct  a 
predictive  model  for  placement. 

Results  showed  that  prediction  of  placement  was  dependent 
on  the  time  frame  under  consideration,  with  models 
differing  completely  for  placement  within  3 months  of 
service,  and  placement  following  3 months  of  service. 

During  the  first  3 months  post-termination,  children's  level 
of  functioning  on  the  CCAR's  socio-legal  domain,  a 
previous  group  home  placement,  and  presence  of  parental 
mental  health  service  needs,  increased  the  odds  of 
placement  (see  Table  2).  The  only  service  variable  which 
was  related  to  placement  was  the  sheer  number  of  follow-up 
services  available  which  reduced  the  risk  of  placement  by 
26%  for  each  service. 

For  the  3-6  month  post-intervention  time  period  (see  Table 
3),  two  variables  were  related  to  reduced  risk  of  placement. 
Families  whose  children  were  at  risk  of  foster  care  (rather 
than  more  intensive  levels  of  care)  were  at  80%  less  risk  of 
placement.  Families  whose  service  focused  on  managing 
child  behavior  were  at  90%  less  risk  of  placement.  In 
contrast,  families  whose  service  focused  on  substance  abuse 
and  very  basic  life-management  skills  were  at  greatest  risk 
(5.7  and  10.0  times  the  risk  of  other  families,  respectively). 

Implications 

Model  Fit:  Implications  for  Program  Decisions 

Information  from  predictive  models  can  be  used  in  two 
ways  at  the  program  level:  (a)  to  make  decisions  about  the 
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ways  at  the  program  level:  (a)  to  make  decisions  about  the 
structure  of  interventions,  and  (b)  to  make  decisions  about 
access  to  services.  In  general,  risk  and  protective  factors, 
and  the  models  built  of  them  are  not  sufficiently  accurate  in 
their  prediction  of  failure  to  warrant  denial  of  access  to 
services.  This  information  is,  however,  useful  in  examining 
the  intervention  so  as  to  structure  it  to  maximize  use  of 
protective  factors  and  craft  interventions  to  focus  on  risk 
factors. 

This  was  certainly  true  for  the  models  which  emerged  from 
this  study.  Although  the  overall  models  for  the  separate  time 
periods  fit  the  data  relatively  well,  they  did  not  predict 
placement  very  well.  In  both  cases,  the  model  did  a good 
job  of  predicting  which  children  would  not  be  placed 
(correctly  classifying  99%  of  these  cases),  but  a very  poor 
job  of  predicting  which  children  would  be  placed.  The  latter 
is,  of  course,  of  the  most  interest  for  restricting  access  to 
services.  The  predictive  model  for  the  3-6  month  time  frame 
was  the  best,  predicting  placement  at  50%  accuracy; 
however,  this  resulted  in  little  practical  usefulness. 

Thus,  these  results  are  useful  in  considering  which  families 
may  be  at  increased  risk  of  placement  for  the  purpose  of 
designing  or  re-designing  the  targeted  intervention  package 
for  families  with  these  characteristics.  These  results  are  not 
useful  in  making  screening  decisions  regarding  access  to 
services. 

Several  findings  are  important  for  program  design.  Clearly 
there  is  need  for  focus  on  short  and  long-term  program 
strategies  for  prolonging  durability  of  the  intervention. 

Many  family  preservation  programs  focus  on  internal  family 
functioning.  However,  programs  must  also  address 
children's  fonctioning  in  the  legal  realm  and  promote  strong 
connections  to  follow-up  services.  As  we  look  at  families 
"down  the  road"  from  intervention,  it  is  clear  that  families 
facing  substance  abuse  and  extreme  family  disorganization 
are  much  less  likely  to  maintain  gains,  which  again  places 
an  emphasis  on  a strong  continuum  of  care  and  on 
intervention  focus  in  these  areas. 

Implications  for  Future  Research 

With  regard  to  the  CCAR,  there  is  need  for  additional  work 
in  specifying  the  structure  of  the  instrument.  There  is  also  a 
need  to  examine  the  concurrent  validity  of  the  instrument  by 
comparing  it  to  other  measures  of  children's  functioning. 
Another  important  area  for  future  research  lies  in  the 
examination  of  the  instrument's  sensitivity  to  change  and  the 
usefulness  of  change  scores  in  assessing  level  of  functioning 
outcomes. 

Additional  qualitative  work  is  needed  on  client  experience 
of  the  helping  relationship.  It  seems  likely  that  much  of  the 
power  of  this  intervention  lies  in  its  approach  to  practice, 
specifically  in  the  nature  of  the  helping  relationship.  If 
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service  interventions  were  adequately  modeled  in 
quantitative  ways,  the  task  of  measuring  the  quality  of  the 
helping  relationship  must  be  faced.  An  approach  to  measure 
development  which  is  rooted  in  the  experience  of  those 
involved  in  the  helping  relationship  is  imperative;  especially 
when  considering  the  experience  of  clients. 

Family  preservation  interventions  occupy  a key  place  in  the 
continuum  of  care.  There  remains  a need  to  move  beyond 
placement  prevention  to  direct  measures  of  client  change, 
including  exploration  of  changes  in  children's  functioning 
and  the  relationships  between  client  change  and  distal 
outcomes. 
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Introduction 


Hawaii's  Healthy  Start  Program  (HSP)  is  a well  established 
screening  and  outreach  program  with  two  components:  (a) 
the  Early  Identification  (EID)  component  which  consists  of 
community-based  screening  to  identify  newborns  at 
environmental  risk  for  child  abuse  and  neglect  and  (b)  the 
Home  Visiting  component  which  consists  of  in-home 
counseling  by  trained  paraprofessionals  to  promote  healthy 
family  functioning  and  child  development.  The  latter 
component  consists  of  role  modeling,  education,  and 
coordinated  linkage  with  pediatric  primary  care  and  other 
needed  community  resources  in  the  child's  first  five  years  of 
life. 


The  HSP  program  philosophy  is  to  target  high  risk  families, 
focusing  on  both  personal  and  environmental  determinants 
of  parenting  behavior,  and  to  promote  competence  by 
"reparenting  the  parents."  The  first  step  is  to  encourage  the 
parent's  emotional  dependence  on  the  home  visitor.  Then, 
over  time,  the  next  steps  are  to  help  the  parents  to  become 
self-sufficient  (Breakey,  Uohara-Pratt,  Morrel-Samuels,  & 
Kolb-Latu,  1991). 

Nationally,  there  is  strong  endorsement  for  home  visiting 
programs  in  general,  and  the  HSP  model  in  particular. 
Several  unresolved  issues,  however,  impede  efforts  to 
establish  community-based  home  visiting  programs:  (a) 
mixed  results  of  past  evaluations,  (b)  limited  study  of 
non-nurse  home  visitors,  (c)  evaluation  of  demonstrations 
rather  than  established  programs,  (d)  little  research  on  the 
types  of  families  most  likely  to  benefit,  and  (e)  uncertain 
cost-benefits  of  home  visiting.  These  issues  render 
evaluation  findings  essential  for  informed  policy  and 
program  development  (United  States  General  Accounting 
Office,  1990). 

This  evaluation  is  a joint  effort  of  the  Johns  Hopkins 
University,  the  Hawaii  State  Health  Department,  and  the 


Hawaii  Medical  Association  (HMA).  Funding  for  the 
evaluation  is  provided  by  the  Robert  Wood  Johnson 
Foundation,  the  Annie  E.  Casey  Foundation,  the  David  and 
Lucile  Packard  Foundation,  the  Federal  Maternal  and  Child 
Health  Bureau,  and  the  Hawaii  State  Department  of  Health. 
Participating  programs  and  agencies  include  Child  and 
Family  Services,  Hawaii  Family  Stress  Center,  Parents  and 
Children  Together,  and  the  Hawaii  Department  of  Human 
Services. 

Method 

The  project  is  being  conducted  over  five  years,  beginning  in 
May,  1994.  The  evaluation  addresses  four  research 
questions: 

• How  closely  does  program  implementation  mirror 
program  design? 

• How  successful  is  the  program  in  achieving  intended 
benefits  for  children  and  families? 

• How  does  fidelity  of  implementation  influence 
program  achievement  of  intended  benefits? 

• How  do  achieved  benefits  compare  to  direct  and 
indirect  program  costs? 

The  evaluation  focuses  on  six  geographically  defined 
communities  on  Oahu  that  are  served  by  HSP  program  sites. 
A family  was  eligible  for  the  evaluation  if  (a)  the  family 
was  eligible  for  the  HSP  itself;  (b)  the  family  was  identified 
as  at-risk  by  an  EID  worker  following  the  usual  HSP 
protocol;  (c)  the  family  was  not  currently  enrolled  in  the 
HSP  for  a prior  birth;  and  (d)  the  mother  did  not  need  a 
translator  (fewer  than  3%  of  those  eligible  for  the  HSP). 

Enrollment  began  on  November  1,  1994  and  continued 
through  December  31,  1995,  achieving  a full  sample  of  685 
participants.  At  each  hospital  on  Oahu,  HSP  early 
identification  workers  (EID)  followed  the  routine  HSP 
screening/assessment  protocol  to  identify  infants  born  into 
at-risk  families.  By  protocol,  all  families  living  in  HSP 
catchment  areas  are  screened  and  assessed.  Those  scoring  ^ 
25  on  Kempe's  Family  Stress  Checklist  are  defined  as 
at-risk.  If  the  family  was  at-risk,  the  EID  worker  described 
the  HSP  and  its  evaluation  according  to  a standardized 
informed  consent  protocol  and  invited  the  mother  to 
participate  in  the  evaluation. 

After  the  mother  gave  informed  consent  for  participation  in 
the  evaluation,  the  EID  worker  called  the  evaluation 
fieldwork  office  to  determine  the  family's  study  group 
assignment.  Families  were  randomly  assigned  to  an  HSP 
intervention  group,  a main  control  group,  or  a testing 
control  group.  Each  intervention  group  family  was  referred 
to  the  Healthy  Start  Program  site  serving  its  community  and 
was  offered  home  visiting  services  following  the  usual  HSP 
protocol.  Each  main  control  group  family  was  also  given  a 
list  of  community  programs. 
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All  families  will  be  followed  for  three  years.  Key  outcome 
variable  indicators  and  measures  to  be  collected  annually 
are  delineated  in  Table  1.  In  addition,  data  on  maternal 
characteristics,  maternal  and  paternal  employment,  maternal 
social  support,  maternal  health  and  psychological 
well-being,  maternal  health  care,  child  health  care,  need  for 
parenting  services,  maternal  and  paternal  substance  abuse, 
paternal  characteristics,  and  family  income  will  be  collected 
on  the  baseline  interview  as  well  as  in  follow-up  interviews. 

Analysis 

The  comparability  of  study  participants  and  non-participants 
will  be  examined  using  standard  techniques  for  the  types  of 
variables  measured  (e.g..  Student's  t-test  for  normally 
distributed  variables,  chi-square  tests  for  binary  data,  etc.) 
for  evaluation  of  all  at-risk  families  at  the  time  of  the  child's 
birth.  In  the  same  way,  the  initial  comparability  of  the  HSP 
and  control  groups  will  be  assessed  for  these  measures  and 
for  baseline  interview  variables.  Overall  process  measures, 
outcome  assessment,  relationship  between  program  process 
and  outcome,  and  cost  benefits  analysis  will  be  examined 
using  standard  techniques  for  the  types  of  variables 
measured. 

Within  both  the  HSP  and  Control  Groups,  families  will  be 
categorized  in  terms  of  characteristics  at  the  time  of  the 
child's  birth:  family  ethnicity,  initial  risk  assessment  score, 
family  substance  abuse,  family  violence,  and  maternal  age. 
For  each  outcome,  multivariable  models  will  be  used  to  test 
for  differences  in  outcome  between  the  HSP  Group  and  the 
Control  Group  in  the  presence  of  differences  in  initial  risk 
and  the  degree  of  resolution  of  other  outcomes. 

Within  both  groups,  levels  of  use  and  associated  costs  will 
be  measured  for  health  services,  child  protective  services, 
police  and  legal  services,  and  other  community  services.  For 
families  in  the  HSP  group,  direct  program  costs  will  be 
measured  as  well.  Tests  for  significant  differences  in  total 
costs  per  child  between  program  and  control  group  families 
will  be  computed  using  t-statistics,  and  controls  for  other 
factors  that  influence  costs  will  be  introduced  via  regression 
analysis. 

Results 

Exploratory  analyses  were  conducted  by  running 
frequencies  on  all  522  variables  for  the  first  664  completed 
cases  in  the  database  examining  distributions  and 
summarizing  responses  in  terms  of  central  tendency  and 
dispersion.  Eighty-two  percent  of  the  families  that  were 
eligible  for  the  evaluation  are  participating  in  the  study.  At 
this  point,  no  differences  in  the  comparability  of  study 
participants  and  non-participants  have  been  detected. 

Comparability  of  Groups 
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Comparison  of  Study  Groups,  the  Healthy  Start  Program 
Group,  and  the  Control  Group  are  comparable  in  terms  of 
demographics  (see  Table  2).  The  only  area  where  p 
approaches  significance  is  in  the  Mother's  Employment  in 
the  Past  Year.  We  will  control  for  this  difference  as  needed 
in  subsequent  analyses. 

Family  Needs 

The  baseline  interview  contains  items  pertaining  to:  (a)  the 
mother's  current  receipt  of  financial,  nutritional  and  housing 
assistance;  (b)  her  perceived  need  for  nutritional,  housing, 
vocational  and  child  care  services;  and  (c)  her  emotional 
health.  Table  3 summarizes  the  percent  of  mothers  in  the 
first  664  cases  analyzed  who  report  using  or  needing 
financial,  nutritional,  housing,  educational,  vocational,  child 
care,  and  mental  health  services.  Family  needs  associated 
with  teenage  childbearing,  domestic  violence,  problem 
alcohol,  and  other  drug  use  are  discussed  below. 

Most  mothers  qualify  for  the  state's  health  coverage 
program,  QUEST,  which  is  targeted  to  low  income  families. 
Over  two-thirds  of  the  mothers  live  in  households  with 
income  below  the  poverty  level,  and  about  two  thirds 
receive  public  assistance  and/or  food  stamps. 

Half  the  mothers  received  WIC  while  pregnant,  and  nearly 
all  plan  to  receive  WIC  postpartum.  Overall,  15%  of  the 
mothers  would  like  to  receive  more  information  on 
breast-feeding.  About  a quarter  of  the  mothers  live  in  public 
housing.  While  virtually  all  live  in  housing  with  complete 
kitchen  and  bathroom  facilities,  1 8%  report  problems  with 
their  current  living  situation.  Also,  13%  feel  they  need  a 
new  place  to  live,  and  9%  feel  they  need  help  securing  new 
living  arrangements. 

Overall,  20%  of  mothers  feel  they  will  need  help  finding  a 
job  or  work  training.  This  is  one-third  of  the  56%  who  plan 
to  return  to  work  in  the  next  year  (not  shown  in  table).  Child 
care  is  another  area  where  a substantial  portion  of  mothers 
need  help.  A total  of  40%  of  the  mothers  report  needing 
help  arranging  child  care  to  return  to  work  or  school. 

Finally,  to  estimate  mental  health  needs,  the  short  form 
version  of  the  RAND  mental  health  measure  was 
administered  (McHomey  & Ware,  1995).  Using  the  cutoff 
suggested  by  RAND  researchers,  95%  of  the  mothers  were 
classified  as  being  in  poor  mental  health.  While  some 
distress,  anxiety,  and  depression  could  be  attributable  to 
pregnancy  and  delivery,  this  figure  is  still  notable. 

To  some  extent,  these  estimates  of  need  may  be 
conservative.  It  is  expected  that  reporting  bias  has  been 
minimized  by  efforts  to  distinguish  the  program  and  the 
evaluation  from  one  another  through  both  an  intensive 
training  of  interviewers  in  neutrality  in  conducting  the 
interview  and  through  careful  wording  of  items  as  well  as 
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interview  and  through  careful  wording  of  items  as  well  as 
use  of  existing,  validated  instruments  where  possible.  Even 
so,  there  is  always  the  possibility  that  some  mothers  will 
give  socially  acceptable  answers.  In  addition,  it  is  possible 
that  mothers  use  different  norms  in  defining  what  they 
consider  a problem,  such  as  a problem  with  housing. 

Family  Characteristics 

Twenty-nine  percent  of  the  families  have  a primary  ethnic 
affiliation  of  Native  Hawaiian,  and  41%  the  families  have 
any  ethnic  affiliation  of  Native  Hawaiian  (see  Table  4V  In 
54%  of  the  first  664  study  families,  the  mother  was  a 
teenager  at  the  birth  of  her  first  child.  Almost  half  (48%) 
report  domestic  violence  between  the  mother  and  father  in 
the  past  year.  P achieves  statistical  significance  in  two  areas: 
^ 3 episodes,  father  toward  mother  and  ^ 3 episodes,  either 
way.  These  numbers  will  continue  to  be  monitored,  and  if 
need  be,  statistical  adjustments  will  be  made  to  compensate 
for  this  difference.  Over  half  (53%)  of  the  families  reported 
problem  alcohol  use,  and  48%  reported  problem  use  of  other 
drugs.  Sixty-eight  percent  of  the  families  reported  problem 
use  of  either  alcohol  or  other  drugs. 

Discussion 

As  previously  noted,  there  is  strong  endorsement  for  home 
visiting  programs  in  general,  and  the  HSP  model  in 
particular.  Rapid  proliferation  of  national  and  international 
replications  of  the  Hawaii  Healthy  Start  Program  make  this 
evaluation  timely.  The  rigorous  methodological  safeguards 
against  bias  will  enable  home-based  interventionists  to  have 
confidence  in  the  study's  findings  at  the  conclusion  of  the 
evaluation.  We  are  now  in  the  midst  of  Year  One  outcome 
data  collection  and  expect  to  begin  reporting  on  program 
impact  in  late  1996. 
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Introduction 


Over  the  past  decade,  child  protective  services  across  the 
country  have  experienced  an  increase  in  reports  of  alleged 
child  abuse  and  neglect.  Researchers  and  others  working  in 
the  child  welfare  field  agree  that  one  of  the  most  important 
factors  in  skyrocketing  protective  services  caseloads  is  an 
increase  in  parental  substance  dependence,  particularly  on 
crack  cocaine  (Curtis  & McCullough,  1993).  The  Children's 
’ Defense  Fund  (1994)  has  found  that  parental  substance 
abuse  is  a factor  in  75%  of  foster  home  placements. 

' Consequently,  substance  abuse  and  dependence  has  become 
' the  "dominant  characteristic"  in  child  welfare  caseloads  in 
twenty-two  states  and  the  District  of  Columbia  (Besharov, 

: 1989). 

While  there  is  widespread  concern  over  the  impact  of 
; parental  substance  dependence  on  the  psychosocial 
development  of  children  and  the  attendant  risk  of  child 
i maltreatment  and  family  breakdown,  little  attention  has 
been  paid  to  developing  and  testing  interventions  to  prevent 
family  deterioration  and  negative  child  outcomes  in  families 
with  a history  of  substance  abuse.  Few  addictions  treatment 
programs  are  aimed  at  the  special  needs  of  parents  with 
alcohol  and  drug  dependence  who  have  children.  Further, 
child  welfare  workers&endash;those.most  likely  to  come  in 
contact  with  parents  with  substance  abuse  issues  and  have 
leverage  to  encourage  treatment&endash;do  not  usually 
possess  the  knowledge  or  training  required  to  intervene 
successfully  with  families  that  are  drug  involved. 

In  an  effort  to  address  the  unmet  needs  of  parents  and 
children  in  families  with  substance  abuse  issues  referred  due 
to  child  maltreatment,  a demonstration  program  was 
developed  in  one  major  metropolitan  area  to  provide 
intensive  in-home  intervention  with  such  families.  Distal  or 
outcome  goals  of  the  program  were  to:  (1)  prevent  further 
child  abuse  or  neglect,  (2)  prevent  family  breakdown  and 
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child  placement,  and  (3)  facilitate  substance  abuse  treatment 
for  caregivers.  Proximal  goals  identified  to  enhance 
achievement  of  the  outcome  goals  included:  (a)  increase 
parenting  skills,  (b)  decrease  levels  of  depression  in 
substance-involved  caregivers,  and  (c)  enhance  the 
cognitive  and  psychosocial  development  of  children. 

Method 

Sample 

Data  were  collected  on  138  participating  families  referred  to 
the  program  by  the  local  public  child  welfare  agency  or 
from  a hospital-sponsored  drug  treatment  center  that  treated 
many  parents  living  in  the  program's  catchment  area. 
Because  of  the  location  of  the  demonstration  program  in  an 
impoverished,  largely  minority  community,  all  participating 
parents  or  caregivers  were  African  American,  most  were 
female,  and  96%  were  receiving  public  assistance.  Less  than 
half  of  the  participants  were  high  school  graduates. 

Intervention 

Prior  research  and  clinical  observation  indicates  that 
intensive  emotional  and  concrete  support  can  enhance  the 
ability  of  parents  with  a history  of  substance  abuse  to  seek 
and  sustain  addictions  treatment  (Nelson-Zlupko,  Dore,  & 
Kauffman,  1 996).  Maintenance  of  sobriety  may  result  in 
improved  parental  mental  health,  thereby  increasing  the 
ability  to  engage  in  positive  parenting,  which,  in  turn, 
decreases  child  maltreatment  and  enhances  child 
functioning. 

Services  provided  in  this  demonstration  program  to  help 
parents  with  a history  of  child  maltreatment  enter  and 
sustain  addictions  treatment  included:  (1)  in-home 
assessment,  supportive  counseling  and  referral  by  a 
substance  abuse  specialist;  (2)  linkages  with  substance 
abuse  treatment  programs;  (3)  developmental  day  care  and 
respite  for  preschool  children  in  referred  families;  (4) 
parenting  education;  (5)  transportation;  and  (6)  emergency 
funds,  clothing,  food,  and  other  forms  of  concrete  support. 

Data  collection 

Data  were  collected  from  multiple  sources  including  (a) 
clinician  observation,  (b)  client  self-report,  (c)  case  record 
review,  and  (d)  external  reporters  such  as  staff  from  drug 
treatment  programs  and  the  day  care  program.  A mix  of 
standardized  instruments  and  project-developed  survey 
forms  were  used.  Standardized  instruments  included  the 
following:  (a)  the  Beck  Depression  Inventory  (BDI),  a 
widely-used  measure  of  depression  which  was  administered 
to  the  focal  parent  at  three  points  in  time;  (b)  the 
Adult-Adolescent  Parenting  Inventory  (AAPI;  Bavolek, 
1984),  a self-report  inventory  designed  to  identify 
inadequate  and  destructive  parenting  attitudes  and 
behaviors;  (c)  the  Denver  Developmental  Screening  Test 
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behaviors;  (c)  the  Denver  Developmental  Screening  Test 
(Frankenberg  & Dodds,  1967),  an  observer  rating  scale  of 
child  developmental  status;  and  (d)  the  Behavior  Checklist 
for  Infants  and  Children  (BCIC;  MacPhee,  Benson,  & 
Bullock,  1988),  designed  to  measure  caregivers'  perceptions 
of  children's  social  relationships,  compliance,  dependency, 
conduct,  and  activity  level. 

Goal  attainment  data  were  extracted  from  the  Family 
Service  Description,  a document  mandated  by  the  public 
child  welfare  agency,  completed  at  intake  and  containing 
case-specific  treatment  goals.  Demographic  data  and 
substance  abuse  history  were  collected  at  intake  using 
project-developed  forms. 

Results 

Intake  Description 

Sixty-four  percent  of  the  parents  in  this  study  identified 
crack  as  their  drug  of  choice.  Twenty-eight  percent 
identified  alcohol,  while  smaller  percentages  reported 
marijuana  or  PCP  as  their  drug  of  choice.  As  has  been 
reported  in  other  studies  of  women  and  substance  abuse,  the 
parents/caregivers  in  this  study  often  used  drugs  in 
combination,  particularly  crack,  alcohol,  and  marijuana. 

At  program  intake,  67%  of  parents/caregivers  were 
accepting  of  the  need  for  substance  abuse  treatment;  5% 
were  willing  to  consider  treatment  but  not  yet  ready  to 
accept  a treatment  referral;  10%  actively  refused  treatment; 
another  1 8%  were  willing  to  participate  in  ongoing 
counseling  or  support  around  their  need  to  accept  treatment. 

Treatment  Completion 

At  three  months  post-program  entry,  the  point  of  the  first 
assessment  of  treatment  progress,  43%  of  parents/caregivers 
were  active  in  an  addictions  treatment  program;  3%  had 
completed  addictions  treatment  and  were  still  active  in  the 
demonstration  program;  28%  had  dropped  out  of  addictions 
treatment  but  were  still  active  in  the  demonstration  program; 
and,  1 8%  had  dropped  out  of  addictions  treatment  and  out 
of  the  program.  At  each  data  collection  point  in  the 
study&endash;three,  six,  and  twelve  months  post-treatment 
entry&endash;4  to  6%  of  clients  who  had  completed 
addictions  treatment  were  known  to  be  substance-involved 
again,  while  another  21  to  31%  who  had  dropped  out  of 
treatment  were  substance-involved. 

Adjustment  and  Skill 

The  BDI  confirmed  what  other  studies  of  this  population 
have  found:  there  are  high  rates  of  depression  among  clients 
of  minority  status  living  in  poverty  who  have  a history  of 
child  maltreatment  (Sachs  & Hall,  1991).  At  program 
intake,  less  than  half  the  parents/caregivers  scored  within 
the  normal  range  on  the  BDI.  Thirty-five  percent  scored 
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the  normal  range  on  the  BDI.  Thirty-five  percent  scored 
within  the  range  for  mild  depression;  1 1%  showed 
mild-to-moderate  depression,  and  6%  were 
moderately-to-severely  depressed.  At  three  months 
post-program  entry,  there  was  substantial,  though  not 
statistically  significant,  improvement  on  the  BDI. 
Seventy-six  percent  of  parents/caregivers  scored  within  the 
normal  range. 

The  AAPT,  a measure  of  parenting  behaviors,  correlated 
highly  with  child  maltreatment,  between  38  and  48%  of 
parents  scored  within  the  problem  range  on  the  four 
subscales  of  this  instrument.  These  findings  are  consistent 
with  other  studies  which  have  found  that  parents  with  a 
history  of  child  abuse/neglect  (a)  often  have  unrealistically 
high  behavior  expectations  for  their  children  (Milner  & 
Chilamkurti,  1991);  (b)  are  less  able  to  empathically 
understand  their  children  (Belsky,  1993);  (c)  more  often 
resort  to  physical  punishment  and  punitive  acts  as  control 
strategies  (Whipple  & Webster-Stratton,  1991);  and  (d)  look 
to  their  children  to  meet  their  own  unmet  emotional  needs. 
At  three  months  post-intake,  the  percentages  of  parents 
scoring  within  the  problem  range  on  the  four  AAPI 
subscales  were  markedly  lower. 

The  Denver  Developmental  Inventory  (Denver  II)  and  the 
BCIC  were  used  to  assess  the  psychosocial  functioning  of 
preschool  children  in  participating  families.  Only  those 
children  who  attended  the  developmental  day  care  program 
were  tested.  At  program  entry,  only  20%  of  children  tested 
within  normal  developmental  limits,  60%  of  the  children's 
scores  indicated  below  normal  developmental  functioning, 
and  results  for  20%  of  this  group  were  questionable  (i.e., 
exhibiting  some  developmental  delays).  By  three  months 
post-entry,  59%  of  children  tested  within  normal  limits  on 
the  Denver. 

Parent  Perspectives 

On  the  BCIC,  which  measures  parents'  perceptions  of  their 
children's  functioning,  between  7 and  18%  of  children  were 
viewed  by  parents  as  problematic  depending  on  the 
dimension  observed.  For  example,  while  only  7%  of 
children  were  seen  as  exhibiting  inappropriate  dependency 
by  parents,  18%  were  viewed  as  too  aggressive,  indicating 
behavior  which  may  be  more  difficult  for  a single  mother 
with  drug  problems  to  manage. 

On  the  goal  attainment  scaling,  agency  caseworkers  rated 
each  family's  progress  on  up  to  five  problems  addressed  in 
treatment  on  a five-point  Likert-type  scale.  At  three  months 
post-entry,  49%  of  problems  addressed  in  treatment  were 
slightly  to  much  better;  51%  were  same  or  worse  as  at 
intake.  By  six  months,  these  figures  were  63%  and  37% 
respectively;  at  twelve  months,  73%  and  27%. 

Day  Care  Program  Comparison 
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Families  who  participated  in  the  day  care  component  of  the 
program  (N  = 71)  had  better  outcomes  than  those  who  did 
not  (N  = 67).  Sixty-five  percent  of  parents/caregivers  of 
children  in  the  day  care  program  completed  addictions 
treatment  as  compared  with  just  16%  of  non-day  care 
parents/caregivers  ( p < .001).  Forty-one  percent  of  day  care 
families  were  discharged  as  meeting  treatment  goals 
compared  to  Just  10%  of  non-day  care  families.  In  addition, 
children  were  removed  to  foster  care  from  34%  of  non-day 
care  families  as  compared  with  23%  of  day  care  families. 
However,  these  last  two  differences  were  not  statistically 
significant. 

Discussion 

Findings  from  this  study  underscore  the  challenges  of 
assisting  parents  who  are  at  high  risk  for  child  maltreatment 
to  accept  and  sustain  addictions  treatment.  While  the 
program  was  successful  in  encouraging  a large  number  of 
parents/caregivers  to  enter  substance  abuse  treatment,  the 
treatment  dropout  rate  was  high,  particularly  among  those 
who  did  not  avail  themselves  of  additional  support  from  the 
day  care  program.  On  the  other  hand,  67%  of  participants 
either  completed  treatment,  or  dropped  out  and  returned  to 
treatment  while  participating  in  the  demonstration  program. 

Current  research  indicates  that  many  people  with  substance 
abuse  issues,  particularly  women,  experience  several 
treatment  episodes  before  maintaining  sobriety.  Participants 
in  this  demonstration  program  confirmed  these  findings. 
Sixty-nine  percent  had  at  least  one  prior  experience  in 
addictions  treatment  and  a high  percentage  had  two  or  more. 
Thus,  even  though  22%  of  clients  (N  = 30)  entered  and 
dropped  out  of  treatment  while  participating  in  this 
demonstration  program,  these  parents/caregivers  are  more 
likely  to  enter  a treatment  program  in  the  future  than  those 
with  no  previous  treatment  experience. 

Program  success  was  not  as  apparent  with  regard  to  the  goal 
of  preventing  family  breakdown  and  preventing  child 
placement.  A child  was  eventually  removed  to  out-of-home 
care  for  28%  of  participating  families  (N  = 30).  There  was  a 
strong  relationship  between  a parent  or  caregiver's  refusal  of 
substance  abuse  treatment  and  a child's  removal  from  the 
home.  It  is  not  uncommon  for  an  intensive  intervention 
program  such  as  this  one  to  identify  cases  of  child  abuse  or 
neglect  that  appear  intractable  to  even  the  most  focused  and 
intensive  intervention.  That  some  parents/caregivers  were 
offered  an  enhanced  array  of  supports  to  facilitate  addictions 
treatment,  yet  were  unable  to  enter  or  sustain  such 
treatment,  suggests  an  unacceptable  level  of  risk  to  children 
in  the  home.  As  has  been  demonstrated  elsewhere, 
parent/caregiver  addiction,  particularly  to  crack-cocaine,  is 
highly  detrimental  to  parenting  functions  (Dore,  Doris,  & 
Wright,  1995). 
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The  theory  imdergirding  the  enhanced  supports  offered 
parents  with  histories  of  substance  abuse  and  child 
maltreatment  in  this  demonstration  program  was  that  by 
increasing  parenting  skills,  decreasing  parent/caregiver 
depression,  and  enhancing  the  cognitive  and  psychosocial 
development  of  children,  the  incidence  of  future  child 
maltreatment  by  participants  could  be  reduced.  As  the 
review  of  findings  indicates,  there  is  evidence  that  parental 
attitudes  and  beliefs  about  both  childrearing  and  levels  of 
parent/caregiver  depression  improved  over  the  first  three 
months  of  active  participation  in  the  program.  Results  on 
the  Denver  II  also  suggest  that,  for  those  children  who 
continued  in  the  day  care  program  over  a three  month 
period,  there  were  significant  developmental  gains.  This 
finding  adds  evidence  that  providing  child  care  is  crucial  to 
enhancing  both  the  probability  that  parents  will  seek  and 
sustain  drug  treatment  and  promote  developmental 
outcomes  for  their  children.  As  current  research  has 
effectively  demonstrated  the  interaction  between  child 
behavior  and  parent  behavior  in  situations  of  child  abuse, 
enhancing  child  functioning  can  only  serve  to  decrease  the 
risk  of  future  maltreatment  of  these  children. 

In  summary,  findings  with  regard  to  effectiveness  of  this 
demonstration  program  cannot  be  stated  with  certainty 
because  this  was  not  a randomized  study  with  a control 
group  design.  However,  given  what  is  Imown  about  treating 
the  population  represented  by  this  sample  of  families  with 
both  substance  abuse  and  child  maltreatment  issues,  this 
program  was  successful  in  helping  some  participants  make 
significant  changes  in  their  lives.  There  is  no  question  that 
the  families  who  participated  in  this  demonstration  program 
present  with  the  most  difficult  problems  facing  families 
today.  Poverty,  lack  of  education  and  employment, 
substandard  housing,  and  family  and  neighborhood 
violence,  were  present  in  addition  to  substance  abuse  for 
nearly  all  participating  families.  The  mix  of  in-home 
counseling  and  support,  linkages  to  addictions  treatment, 
developmental  day  care,  parenting  education,  transportation, 
and  concrete  services  may  be  positive,  effective 
combination  in  addressing  the  growing  social  problem  of 
child  maltreatment  stemming  from  parental  substance 
abuse. 
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Recent  research  indicates  that  social  support  is  an  important 
factor  in  an  individual's  mental  health  and  interpersonal 
adjustment  (Frecknall  & Luks,  1992;  Queen,  1994). 
Traditionally,  family  and  close  friends  served  as  social 
networks  that  provided  the  individual  protection  against 
stress  or  adversity.  Adolescents  who  demonstrate  positive 
adjustment  to  ongoing  stress  often  identify  a relative,  friend, 
or  neighbor  as  a natural  mentor  who  contributed  to  their 
success  (Rhodes,  Contreras,  & Mangelsdorf,  1994).  Rhodes, 
Ebert,  and  Fischer  (1992)  have  shown  that  a mentor 
relationship  helps  adolescent  mothers  experience  less 
depression  and  feel  less  distressed  by  interpersonal 
difficulties.  Based  on  such  research,  it  seemed  reasonable  to 
hypothesize  that  mentor  relationships  could  have  a positive, 
therapeutic  effect  on  children  and  adolescents  at  significant 
risk  for  out-of-home  placement. 

Belief  in  the  integrity  and  healing  capacity  of  the  family  unit 
coupled  with  the  soaring  cost  of  out-of-home  placements 
has  led  Waukesha  County,  Wisconsin  to  develop  creative, 
collaborative  efforts  to  maintain  troubled  children  in  their 
homes.  The  Clinical  Services  Division  of  the  Waukesha 
County  Health  and  Human  Services  Department  developed 
the  Mentor  Services  Program  in  1991  based  on  a 
community  needs  assessment  that  identified  service  gaps  in 
the  areas  of  social  supports  and  respite  care  for  children 
with  severe  emotional  disturbance  (SED).  Developed  as 
placement  prevention.  Mentor  Services  expand  the  array  of 
1 clinical  treatment  resources  and  community-based  care 
I received  by  the  child  and  family  and  can  be  tailored  to 
I individual  needs.  The  program  is  delivered  through  a 
i services  contract  with  the  outreach  department  of  a 
j residential  treatment  center.  The  center  staff  recruit  and  hire 
j mentors  in  part-time  positions,  matching  the  training,  skills, 

I and  availability  of  the  mentor  to  the  specific  needs  and 
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strengths  of  the  participating  child  and  family. 

Mentors  provide  regular,  consistent,  face-to-face  and 
telephone  contact  with  children  who  have  severe  emotional 
disturbance.  Mentoring  is  part  of  a coordinated  team 
treatment  effort  to  maintain  and  support  children  in  their 
home  cornmunities  by  strengthening  their  family 
relationships  and  by  facilitating  their  successful  integration 
into  existing  community  resources. 

Mentors  provide  support  and  respite  care  for  children  and 
their  parents  through  monitoring  and  supervision  of  the 
child's  functioning  in  various  settings,  both  recreational  and 
educational  (e.g.,  modeling  and  coaching  of  various  social 
skills  including  sharing,  sportsmanship,  and  anger  control 
techniques;  supportive  counseling  and  utilization  of  child 
behavior  management  strategies;  and  assistance  to  families 
in  child/parent  conflict  resolution).  When  appropriate  and 
necessary,  mentors  provide  assistance  to  schools  by 
monitoring  and  addressing  barriers  to  school  success. 
Mentors  provide  verbal  reports  at  least  weekly  to  the 
referring  worker  and  attend  treatment  team  meetings  to 
evaluate  the  child's  progress.  A written  record  of  all  contacts 
with  the  child/family  is  maintained,  and  a monthly  summary 
is  prepared  regarding  the  therapeutic  connections  between 
the  activities  and  the  ongoing  treatment  goals. 

Method  and  Results 

In  this  study  thirty  children  meeting  SED  criteria  and 
considered  to  be  at  extreme  risk  for  out-of-home  placement 
were  enrolled  in  the  mentor  service  for  five  hours  each  per 
week.  All  children  carried  a DSM-IV  psychiatric  diagnosis 
(American  Psychiatric  Association,  1994),  and  according  to 
county  criteria  for  placement,  would  have  been  expected  to 
be  placed  in  an  out-of-home  setting  within  six  months  of 
their  original  SED  staffing.  Sample  demographics  are 
contained  in  Table  1.  Of  the  30  children  discharged  from  the 
program  during  1991-95,  22  remained  in  their  family  home 
six  months  post  discharge.  Five  of  the  children  had  been 
placed  into  residential  treatment  and  three  had  moved  from 
Waukesha  County's  jurisdiction.  This  placement  rate  of 
18.5%  is  significantly  less  than  the  placement  rate 
experienced  by  a control  group  of  ten  children  who  were 
referred  for  mentoring  service  but  not  served.  The 
placement  rate  for  the  control  group  was  50%  within  six 
months  of  referral. 

The  average  monthly  cost  for  all  treatment  intervention  with 
the  mentored  children  was  approximately  $2,300  per  month, 
with  mentor  costs  of  about  $430  per  month.  This  compares 
to  an  average  cost  of  approximately  $4,500  per  month  for 
residential  care  and  in  excess  of  $10,700  per  month  for  a 
State  mental  health  institution  (see  Figure  11. 

Parents  completed  7-point  Likert-type  scales  on  nine 
parameters  designed  to  measure  program  effectiveness  in 
reducing  home,  school,  and  community  problems.  Parents 
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reducing  home,  school,  and  community  problems.  Parents 
also  completed  ten  scales  designed  to  measure  their 
satisfaction  with  program  characteristics  such  as 
child/mentor  match,  length  of  service,  and  respite  (see 
Figure  2y  The  results  suggest  that  families  viewed  mentor 
services  as  an  integral  and  effective  component  of  the 
treatment  process.  Parental  ratings  indicate  that  mentoring 
had  the  largest  initial  impact  on  problems  and  conflicts 
within  the  home.  Overall,  changes  associated  with 
mentoring  services  were  relatively  stable  over  the  six 
months  following  discharge  from  the  program.  Scale  values 
for  acting  out  behavior  (e.g.,  alcohol  or  drug  abuse  truancy, 
legal  problems,  etc.)  showed  the  least  deterioration  over  the 
six  month  span.  Parents  also  rated  the  program  highly  as  a 
respite  service  for  family  members,  including  the  child. 

Discussion 

Placement  data  suggest  that  mentoring  can  be  a highly 
effective  clinical  tool  for  certain  populations  of  at-risk 
children.  Results  show  that  clinical  mentoring  may  be  more 
effective  when  the  child  carries  a diagnosis  of  a disruptive 
behavior  disorder  compared  with  those  children  diagnosed 
with  an  affective  disorder  such  as  major  depression.  This 
may  be  because  the  major  affective  disturbances  disrupt  the 
child's  ability  to  relate  socially  and  form  meaningful 
interpersonal  relationships.  The  ability  to  form  such 
relationships  is  believed  to  be  the  foundation  of  therapeutic 
mentoring.  This  review  also  indicates  that  mentoring  is  less 
successful  with  children  under  the  age  of  12  or  13,  possibly 
because  younger  children  have  developmentally  less 
sophisticated  interpersonal  skills  and  have  more  difficulty 
forming  a social/therapeutic  relationship. 

When  compared  to  the  control  group  of  non-mentored 
children,  mentored  children  were  significantly  less  likely  to 
be  placed  in  a child  caring  institution,  such  as  a residential 
treatment  center,  group  home,  or  treatment  foster  care. 
Mentoring  was  also  an  extremely  cost  effective  component 
of  overall  treatment  for  at-risk  children  when  compared  to 
those  more  traditional  treatment  venues  (see  Fimire  IV 
Families  consistently  rated  mentoring  as  an  integral  and 
crucial  resource  in  the  individualization  and  tailoring  of  a 
collaborative  plan  of  care  among  home,  school,  and 
community  services.  Lastly,  families  continually 
emphasized  the  importance  and  value  of  the  respite  inherent 
in  ongoing  mentor  services. 
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Introduction 


The  majority  of  treatment  outcome  studies  in  outpatient  children's  mental  health 
services  have  taken  place  in  research  clinics.  Findings  from  research  studies  have 
been  consistently  positive.  In  contrast,  there  exist  only  a handful  of  published 
outcome  studies  based  in  "naturalistic"  settings  with  mixed  and  far  less  positive 
results;  Weisz  and  Weiss  (1993)  address  this  problem  with  the  "good  news  bad 
news  joke"  which  states  that  "the  good  news  is  that  child  psychotherapy  works;  the 
bad  news  is,  not  in  real  life"  (p.  96). 

Contrasting  procedures  between  research  and  "real  world"  clinics  may  explain  these 
observed  differences  in  treatment  outcome.  Werry  and  Andrews  (1996)  point  out 
that  "there  is  little  resemblance  between  what  researchers  study  and  what 
practitioners  do"  (p.  879).  Differences  in  procedures  between  the  research  and 
naturalistic  settings  include  the  following:  subject  recruitment,  parental  involvement 
in  treatment,  stringent  treatment  protocols,  exclusionary  criteria,  manualized 
treatment,  and  completion  of  measures  by.both  therapists  and  patients  (Kazdin, 

1991 ; Kendall  & Southam-Gerow,  1995).  Such  procedures  are  often  an  anathema  to 
both  clients  and  clinicians  in  naturalistic  settings  and  likely  result  in  different 
clinicians  treating  and  different  subject  populations  being  seen  at  research  versus 
non-research  based  clinics. 

In  the  adult  literature,  Seligman  (1995)  describes  this  dialectic  by  distinguishing 
between  "efficacy"  studies  in  research  clinics  and  "effectiveness"  studies  in 
"naturalistic"  settings.  Under  tightly  controlled  conditions  with  sophisticated 
methodology,  efficacy  studies  conducted  in  research  clinics  differ  from 
effectiveness  studies  which  assess  "how  patients  fare  under  the  actual  conditions  of 
treatment  in  the  field"  (Seligman,  1995,  p.  966).  Although  both  approaches 
contribute  to  our  knowledge  of  treatment  outcome,  "effectiveness"  studies  may  hold 
the  greatest  potential  to  teach  us  about  the  majority  of  clients  and  clinicians  who 
participate  in  the  "real  world"  of  mental  health.  Seligman  (1995)  further  extends  this 
contention  by  asserting  that  "the  efficacy  study  is  the  wrong  method  for  empirically 
validating  psychotherapy  as  it  is  actually  done"  (p.  966).  However,  a number  of 
investigators  in  children's  mental  health  have  proposed  research  methods  to  close 
this  chasm.  They  have  underscored  the  need  to  examine  ways  to  improve  the 
transition  from  efficacy  research  to  effectiveness  research  (Clark,  1995),  increase 
the  transportability  of  research  based  treatment  to  naturalistic  settings  (Kendall  & 
Southam-Gerow,  1995),  and  bridge  the  gap  between  laboratory/university  and 
clinic/community  based  treatment  (Henggeler,  Schoenwald,  & Pickrel,  1995; 
Hoagwood,  Hibbs,  Brent,  & Jensen,  1995;  Weisz,  Donenberg,  Han,  & Weiss,  1995). 

The  present  study  was  designed  to  contribute  to  the  scant  literature  on  outcome  data 


derived  from  naturalistic  settings.  It  was  hypothesized  that  children  would 
demonstrate  reliable  and  positive  improvement  from  pre-  to  post-treatment  based  on 
standard  parent  and  therapist  outcome  measures.  Specifically,  parent  rated 
improvement  of  their  children's  internalizing  and  externalizing  symptoms  on  the 
Child  Behavior  Checklist  (CBCL;  Achenbach  & Edelbrock,  1983)  and  therapist 
rated  improvement  on  both  the  Global  Assessment  of  Functioning  (GAF;  Diagnostic 
and  Statistical  Manual  of  Mental  Disorders,  1987)  and  the  Children's  Global 
Assessment  Scale  (CGAS;  Shaffer  et  al.,  1983)  were  expected.  Furthermore,  given 
an  emphasis  in  treatment  on  parent  involvement,  it  was  hypothesized  that  families 
would  report  higher  levels  of  family  cohesion  and  organization,  less  family  conflict, 
and  a greater  sense  of  community  involvement  (e.g.,  recreation)  as  assessed  by  the 
subscales  of  the  Family  Environment  Scale  (FES;  Moos  & Moos,  1986). 

Method 

Subjects 

Parents  and  clinicians  of  fifty-one  children  (N  = 51;  32  boys/19  girls),  receiving 
between  16  and  90  therapy  sessions,  completed  measures  before  initiation  of 
treatment  and  at  termination.  All  cases  represent  agreed-upon  discharge  between  the 
family  and  therapist.  Demographic  variables  are  presented  in  Table  1.  This  sample 
represents  a fraction  (less  than  5%)  of  the  clinic  population  over  a 5-year  period 
(1989-1994)  and  contains  an  over-  representation  of  Caucasian  children  (80%), 
cases  that  were  self-referred  (94%),  and  cases  with  significantly  more  treatment 
sessions  (M  = 43)  in  comparison  to  our  general  clinic  population.  During  the  time 
period  that  this  data  was  collected,  Caucasian  children  represented  40%  of  the  clinic, 
only  20%  of  the  children  were  self-referred,  and  cases  were  seen  an  average  of  24 
sessions.  However,  the  current  subject  sample  did  not  differ  from  the  general  clinic 
pool  from  which  it  came  in  age,  gender,  caretaker  status  (one  parent  versus  two 
parent),  residence  (urban  versus  suburban),  socioeconomic  status  as  measured  by  the 
Hollingshead  (1975)  Five-factor  Index  of  Social  Status,  or  pay  source  (insurance 
versus  medicaid). 

Measures 

The  parent  rated  CBCL  includes  1 1 8 items  on  a 0-2  point  scale  which  rate  a variety 
of  behavior  problems  and  yields  an  internalizing,  externalizing,  and  total  score 
composite.  The  FES  includes  90  tme-false  items  about  family  functioning  across  10 
subscales  (i.e.,  conflict,  cohesion,  expressiveness,  independence,  achievement, 
intellect,  religious,  recreation,  organization,  & control).  The  GAF  measures  overall 
adaptive  functioning  and  is  rated  by  clinicians  on  a 100  point  scale  (higher  scores 
indicate  better  functioning).  Similarly,  the  CGAS  is  rated  by  clinicians  on  a 100 
point  scale  and  measures  the  child's  lowest  level  of  functioning  within  the  past  week 
(higher  scores  indicate  better  functioning). 

Procedures 

All  clients  were  seen  in  the  Outpatient  Clinic  of  the  Yale  Child  Study  Center.  The 
clinic's  orientation  is  primarily  long-term,  psychodynamic  treatment.  Prior  to  clinic 
contact,  parents  completed  both  the  CBCL  and  FES;  these  two  measures  were  also 
completed  by  parents  at  discharge.  The  treating  clinician  rated  the  child  on  the  GAF 
and  CGAS  at  evaluation  and  discharge. 

Results 

Main  Analyses 
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To  test  our  general  hypothesis  that  children  would  improve  as  a result  of  therapy,  we 
ran  paired  sample  t-tests  on  our  primary  criterion  measures  of  interest  (i.e.,  the 
parent  completed  CBCL  and  FES  scales  and  the  clinician  rated  GAF  and  CGAS).  In 
order  to  control  for  chance  findings,  we  used  the  conservative  Bonferonni  correction 
and  required  a value  of  p = .006  (i.e.,  .05/8)  to  consider  our  findings  significant.  At 
this  adjusted  alpha  level,  significant  improvement  was  reported  on  the  CBCL 
(internalizing,  externalizing,  and  total  scales),  the  CGAS,  and  GAF.  In  addition, 
there  was  a significant  increase  in  the  "Recreation"  and  "Organization"  subscales  of 
the  FES  (see  Table  2).  There  was  no  significant  change  from  pre-  to  post-treatment 
on  the  "Conflict"  and  "Cohesion"  subscales  of  the  FES. 

Discussion 

Fifty-one  children  receiving  long  term  dynamically  oriented  therapy  at  a "real 
world"  clinic  improved  in  their  fimctioning  as  rated  by  parents  and  clinicians.  In 
addition,  parents  reported  improved  family  management  practices  and  a decrease  in 
social  isolation  from  the  beginning  to  end  of  treatment.  These  parallel  gains  in  child 
and  family  functioning  address  the  interactions  between  child  and  family  well-being 
and  are  consistent  with  prior  research  (Armbruster,  Dobuler,  Fischer,  & Grigsby, 
1996). 

Despite  these  positive  results,  the  validity  of  our  findings  is  limited  by  a number  of 
factors.  First,  without  a control  group  it  is  difficult  to  attribute  improvement  to  the 
intervention  or  to  natural  maturational  processes.  The  effect  of  maturation  is 
particularly  relevant  in  view  of  an  average  treatment  duration  of  nearly  a year. 
Second,  although  the  sample  was  similar  to  the  general  clinic  population  in  age, 
gender,  caretaker  status,  residence,  pay  source,  and  SES,  it  differed  in  that  it  was 
comprised  primarily  of  Caucasian,  self-referred,  long  term  therapy  cases.  Finally, 
only  families  who  were  discharged  from  treatment  by  mutual  agreement  between 
therapist  and  client  and  completed  the  post  measures  were  included  in  the  study. 
Excluded  were  those  families  who  dropped  out  of  treatment  or  were  discharged  but 
who  did  not  complete  the  post  measures.  Follow-up  for  these  families  was  beyond 
the  scope  of  this  study. 

These  limitations  may  be  understood  in  a number  of  ways.  The  preponderance  of 
Caucasian  families  in  this  study  may  point  to  the  issue  of  cultural  congruity  between 
therapist  and  client  (Armbruster  & Kazdin,  1 994).  Studies  have  shown  that  similar 
ethnicity /race  of  clinician  and  client  predict  continuance  in  therapy  (Cheung  & 
Snowden,  1990;  Flaskerud,  1986).  At  the  time  of  this  study,  there  was  a significant 
lack  of  minority  therapists  at  the  clinic  which  may  have  interfered  with  African 
American  families  forming  positive  alliances  with  their  therapists.  The  group  for 
which  pre-  and  post-  measures  were  collected  may  reflect  a "goodness  of  fit"  (Chess 
& Thomas,  1986)  between  family  and  therapist.  This  "fit"  may  be  partially 
accounted  for  by  compatible  levels  of  motivation,  accountability  and  commitment, 
as  well  as  similar  racial/ethnic  characteristics.  The  motivational  factor  is  further 
supported  by  the  fact  that  the  majority  (94%)  of  families  were  self-referred. 

In  this  study,  self-selection  bias  in  clinician  and  parent  may  lead  us  to  conclude  that 
treatment  is  effective,  when  in  fact,  we  have  sampled  only  a small  percentage  of 
highly  motivated  parents  and  clinicians.  This  study  underscores  difficulties  inherent 
in  evaluating  effectiveness  in  "naturalistic"  settings  where  clinicians  are  less 
motivated  to  adhere  to  research  designs,  especially  those  involving  the 
administration  of  post  measures  (Henggeler  et  al.,  1995;  Hoagwood  et  al.,  1995; 
Kazdin,  1991).  The  advent  of  managed  care  with  its  requirements  for  accountability 
(e.g.,  treatment  outcome  data)  may  provide  the  opportunity  to  conduct  research  in 
"naturalistic"  settings  with  a more  representative  sample.  In  addition,  managed  care 
will  undoubtedly  restrict  the  length  of  treatment  and  reduce  the  likelihood  that 
improvement  is  due  solely  to  maturational  effects.  Future  studies  will  be  able  to 
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improvement  is  due  solely  to  maturational  effects.  Future  studies  will  be  able  to 
examine  populations  within  this  modified  framework. 

In  summary,  "naturalistic"  (Weisz  & Weiss,  1993)  or  "real  world"  (Jensen,  Bloedau, 
& Davis,  1990)  child  outpatient  mental  health  clinics  represent  the  majority  of 
mental  health  services  provided  to  children.  Our  findings  suggest  that  reliable  and 
positive  results  can  be  achieved  for  those  families  and  therapists  in  "real  world" 
clinics  who  remain  committed  and  accountable  to  the  therapeutic  process.  Although 
this  is  an  initial  study  with  limitations,  it  offers  an  optimistic  glimpse  into  the 
outcomes  achieved  by  "treatment  as  usual." 
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Introduction 


The  Fort  Bragg  demonstration  provided  a comprehensive 
approach  to  the  delivery  of  mental  health  and  substance 
abuse  services  to  a population  of  approximately  48,000 
(FY95)  military-related  children  under  18  years  of  age  who 
resided  within  the  Fort  Bragg  catchment  area.  A major 
purpose  of  the  project  was  to  study  the  implications  of 
expanding  the  health  care  benefit  package  provided  to 
military  families  through  the  Civilian  Health  and  Medical 
Program  of  the  Uniformed  Services  (CHAMPUS)  by 
providing  a full  continuum  of  mental,  health  and  substance 
abuse  services  (Bickman  et  ah,  1995). 

As  an  experiment  in  systems  change,  the  Fort  Bragg 
demonstration  yields  a wealth  of  valuable  "lessons  learned" 
for  policymakers,  practitioners,  researchers,  and  consumers. 
These  lessons  have  implications  both  for  those  involved  in 
children's  systems  reform  efforts,  as  well  as  those 
undertaking  managed  care  initiatives  affecting  children  and 
their  families. 
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Method 


Over  the  last  six  months  of  the  demonstration,  over  two 
hundred  individuals  were  interviewed,  including 
administrators,  clinicians,  families,  policymakers,  and  other 
stakeholders,  who  had  been  involved  in  the  Fort  Bragg 
project  since  its  inception.  Interviews  were  conducted  by  a 
team  of  independent  policymakers,  researchers,  family 
members,  and  clinicians,  all  of  whom  have  been  involved  in 
national  and  state  systems  reform  issues  over  the  last 
decade.  The  team  also  reviewed  the  extensive  written 
documentation  available  from  the  demonstration,  the  formal 
evaluation  conducted  by  Vanderbilt  University,  and  client 
characteristic  and  service  data  over  the  life  of  the  project. 

What  Fort  Bragg  Was  and  Was  Not 

Through  the  study,  there  emerged  greater  clarity  as  to  what 
the  Fort  Bragg  demonstration  was  and  was  not. 

Fort  Bragg  was  not  managed  care.  The  demonstration 
sought  specifically  to  increase  access  and  utilization  without 
particular  regard  to  cost,  in  contrast  to  managed  care,  which 
is  concerned  fundamentally  with  controlling  cost  by 
managing  access  and  utilization.  There  was  no  risk  attached 
to  management  and  service  contracts  (which  were 
cost-reimbursable);  many  incentives  existed,  on  both  the 
supply  and  demand  side,  to  provide  and  use  services. 

The  demonstration  also  was  not  a system  of  care;  that  is,  it 
was  not  an  integrated  continuum  of  mental  health  and 
related  services  and  supports  in  which  there  is  shared 
"ownership"  of  the  system  across  multiple  child-serving 
systems  (i.e.,  education,  child  welfare.  Juvenile  Justice  and 
mental  health). 

Fort  Bragg  was  not  a continuum  for  "high  end  users"  (i.e., 
children  with  serious  disorders,  only).  Nearly  60%  of  the 
evaluation  sample  received  no  service  more  intensive  than 
outpatient. 

Fort  Bragg  was  a demonstration  of  a community-based 
continuum  of  mental  health  and  substance  abuse  services, 
with  a single  point  of  access,  for  a total  child/adolescent 
eligible  population  (i.e.,  for  both  acute  and  extended  care 
populations  and  for  children  with  both  mild  and  serious 
disorders).  It  changed  an  existing,  poorly  regarded  system, 
as  it  intended;  it  expanded  the  array  of  services,  eliminated 
co-pays  and  deductibles,  reduced  use  of  inpatient  and 
residential  treatment,  increased  access  and  utilization,  and 
was  held  in  generally  high  regard  by  families  and  the  larger 
community.  With  this  success,  however,  there  emerged 
countervailing  pressures  and  issues  (e.g.,  cost  pressures, 
overutilization  of  multiple  services,  unrealistic  expectations 
on  the  part  of  some  families  and  clinicians,  and  apparent 
cost  shifting  to  and  from  the  demonstration).  To  its  credit, 
the  demonstration  made  every  effort  to  be  a self-correcting 
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the  demonstration  made  every  effort  to  be  a self-correcting 
experiment. 

Critical  Lessons  Learned 

Service  Utilization 

At  least  two  important  lessons  emerge  from  looking  at  the 
greatly  expanded  service  utilization  created  by  the 
demonstration.  The  first  is  the  "if  you  build  it,  they  will 
come"  lesson.  If  mental  health  services  are  offered  to 
families  in  ways  that  are  accessible,  stigma-reducing,  and 
individualized,  they  will  be  used&endash;and  probably 
more  heavily  than  a developing  system  can  accommodate 
without  creating  problems  in  other  areas,  such  as  cost 
problems,  inadequate  provider  training,  inadequate 
monitoring,  and  quality  assurance.  A response  to  these 
problems  is  not  necessarily  to  restrict  access,  which  remains 
a worthwhile  policy  goal,  but  rather  to  design  the 
demonstration  and  manage  services  during  implementation 
in  ways  that  promote  more  deliberative  start-up  and  efficient 
treatment. 

A second  lesson  from  looking  at  service  utilization  in  Fort 
Bragg  is  the  importance  of  analyzing  as  accurately  as 
possible  the  population  eligible  for  services  prior  to 
implementation  (i.e.,  who  and  how  many  can  be  expected  to 
use  services).  This  requires  striking  some  balance  between 
using  prior  utilization  and  prevalence  data.  The  Fort  Bragg 
demonstration  was  hampered  by  the  inadequacy  of  the 
Army's  data  regarding  how  many  children  were  using 
services  and  the  types  of  services  being  used,  as  well  as  the 
insistence  to  base  expected  use  on  prior  utilization  only. 
There  is  a particular  lesson  here  for  managed  care  initiatives 
which  have  a tendency  to  rely  on  prior  utilization  to  develop 
capitation  rates.  The  Fort  Bragg  project  suggests  strongly 
that  use  of  prior  utilization  alone  will  be  an  inaccurate 
predictor  if  it  is  applied  to  a more  attractive,  accessible, 
community -based  continuum  of  behavioral  health  services. 

Fort  Bragg  points  to  the  need  for  individualized  system 
planning  for  different  subclusters  of  children  in  a total 
eligible  population.  For  example,  while  "high  end"  users 
may  need  intensive  case  management  and  treatment 
planning,  this  is  clearly  not  the  case  for  children  using  only 
outpatient  services.  Children  involved  in  other  public 
systems,  particularly  child  welfare,  will  require  more  time 
on  the  part  of  clinicians  to  manage  the  boundaries  between 
the  systems  and  deal  with  the  requirements  of  the  other 
system.  Children  with  ADHD  may  require  less  intensive 
involvement  with  mental  health  but  greater  collaboration 
with  the  pediatric  services.  Fort  Bragg  suggests  an 
important  lesson  about  the  need  to  desegregate  the  eligible 
population  into  types  of  users  and  institute  approaches  that 
m^e  sense  for  different  subpopulations&endash;the  "not 
every  child  needs  the  same  system"  lesson. 

Cost 


52 


Perhaps  the  most  important  cost-related  lesson  from  Fort 
Bragg  is  that  of  linking  cost  and  quality  issues  in  design  and 
implementation&endash;a  lesson  which  Fort  Bragg  itself 
learned,  somewhat  painfully,  and  took  steps  to  change.  It  is 
not  possible  to  maintain  quality  of  care  without  paying 
attention  to  cost;  clinical  viability  alone  is  not  sufficient. 
Inevitably,  costs  will  run  high  if  not  attended  to  from  the 
outset,  leading  to  enormous  pressures  for  cost  containment 
that  then  can  jeopardize  quality  of  care.  Cost  containment 
pressures,  in  turn,  will  aggravate  tendencies  of  cost-shifting 
to  other  child-serving  systems. 

A related  cost  lesson  is  the  importance  of  thinking  through 
in  advance  the  effect  on  supply  and  demand  of  the 
incentives  that  are  built  into  the  system  for  providers  and 
consumers  to  provide  and  use  services,  respectively.  To  its 
credit.  Fort  Bragg  took  a number  of  steps  by  the  latter  half 
of  the  demonstration  (post  the  evaluation  phase)  to  modulate 
some  of  these  incentives  and  control  costs  without 
necessarily  jeopardizing  quality  of  care.  It  also  instituted 
greater  efficiencies  in  service  delivery  through  the 
following:  (a)  streamlining  its  intake  process;  (b) 
emphasizing  shorter-term,  problem  solving  therapies;  (c) 
adhering  more  closely  to  level  of  care  criteria;  (d)  instituting 
guidelines  for  use  of  psychiatrists;  (e)  streamlining  its  case 
management  process;  and  (f)  streamlining  the  treatment 
planning  process.  It  created  more  structured  utilization 
review,  with  a clearer  focus  on  length  of  stay,  and  imposed  a 
more  integrated  focus  on  cost,  quality,  and  service  issues 
through  its  quality  improvement  process  with  feedback 
loops  to  service  providers  about  both  cost  and  quality.  In 
this  process,  the  demonstration  also  had  to  begin  to  clarify 
its  own  values  about  cost  and  quality  issues&endash;again, 
an  important  lesson.  When  cost  containment  values  were 
introduced  into  the  service  system  late  in  the  demonstration, 
there  was  indeed  a dramatic  reduction  in  the  cost  of 
providing  treatment. 

A strong  argument  can  be  made  from  the  Fort  Bragg 
experience  that  research  and  development  costs  and  costs 
associated  with  requirements  imposed  by  demonstration 
sponsors  should  be  budgeted  and  tracked  separately  from 
the  costs  associated  with  service  delivery.  Fort  Bragg  also 
suggests  (as  do  many  other  demonstrations  in  the  children's 
arena)  that  the  time  normally  accorded  to  systems  change 
efforts  (typically,  three  to  five  years)  is  simply  too  short  to 
preclude  inefficiencies  in  start-up,  development,  and 
modification  of  new  approaches  (Friedman,  1996). 

A final  cost  lesson  from  Fort  Bragg  relates  to  the  possibility 
that  greater  efficiencies  may  very  well  have  been  achieved, 
at  least  with  respect  to  Fort  Bragg  children  involved  in 
multiple  systems,  had  the  demonstration  actually 
implemented  a system  of  care  with  shared  ownership  across 
the  major  child-serving  systems,  instead  of  a continuum  of 
mental  health  services  "owned"  by  the  Army.  Certain 
inefficiencies  occurred  because  separate  systems  prevailed. 
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inefficiencies  occurred  because  separate  systems  prevailed. 

Service  Delivery 

A great  many  lessons  can  be  drawn  from  the  demonstration 
about  effective  service  delivery. 

Regarding  intake  and  assessment: 

• Intake  staff  need  access  to  an  array  of  crisis  options; 

• intermediate-level  services,  particularly  in-home 
services,  need  to  be  available  from  the  outset  to 
prevent  inappropriate  hospitalization  and  residential 
care; 

• an  intake  process  that  focuses  on  presenting  problems, 
using  a problem  list,  rather  than  on  "nailing  a 
diagnosis"  is  needed  to  promote  efficiency; 

• small  co-pays  are  needed  to  prevent  an  initial 
recurring  problem  of  missed  appointments;  and 

• intake  staff  need  to  be  trained  well  in  the  treatment 
philosophy  and  goals  of  the  demonstration,  how  the 
service  system  is  organized,  and  the  resources 
available  to  them. 

Regarding  the  treatment  team  process: 

• It  is  critical  to  make  the  treatment  team  responsible 
for  managing/monitoring  length  of  stay  and 
appropriateness  of  care  to  better  link  treatment  and 
cost  concerns; 

• a multidisciplinary  treatment  team  does  not  have  to  be 
held  for  every  child  entering  the  system  but  should  be 
reserved  for  children  with  serious  or  complex 
disorders; 

• the  more  people  who  are  involved  in  the  treatment 
team  process,  the  more  time-consuming  and 
expensive  it  will  be  (i.e,  multiple  players  on  the 
treatment  team  may  aggravate  the  problem  of  overuse 
of  multiple  services,  since  each  player  has  a tendency 
to  become  an  advocate  for  the  service  he  or  she  is 
representing,  which  also  drives  up  costs); 

• an  effective  treatment  team  process  is  one  governed 
by  tight  clinical  leadership  and  a better  understanding 
of  how  the  various  levels  of  care  relate  to  one  another; 

• much  more  needs  to  be  learned  about  what  service 
components  work  best  for  which  types  of  children 
over  what  period  of  time;  and 

• it  is  important  to  be  creative  in  treatment  planning  and 
not  allow  a child  to  stay  in  an  inappropriate  level  of 
care  simply  because  the  needed  service  is  unavailable. 

Quality  Improvement 

A number  of  lessons  can  be  drawn  about  quality 
improvement  (QI).  The  QI  program  should  be  developed 
with  the  involvement  of  staff  and  providers,  as  well  as 
families,  and  incorporate  peer  review  to  create  ownership 
and  "buy-in."  QI  should  provide  information  to  clinicians 
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and  "buy-in."  QI  should  provide  information  to  clinicians 
concurrently  with  service  provision,  not  retrospectively,  so 
that  feedback  in  areas  such  as  length  of  stay  and  family 
satisfaction  gets  to  providers  in  a timely  fashion,  not  after 
the  fact.  QI  should  track  more  than  cost  and  utilization;  it 
needs  also  to  track  systems  outcomes  across  the  continuum, 
client  outcomes,  and  family  and  youth  satisfaction.  QI  and 
utilization  review  functions  need  to  be  integrated.  QI  data 
needs  to  be  tied  to  credentialling  of  providers.  It  is 
unrealistic  to  expect  a full-blown  QI  program  to  be  in  place 
before  clinical  services  are  fully  developed.  Implementation 
of  an  effective  QI  program  is  a developmental  process  that 
requires  time. 

Staff  and  Provider  Network 

One  of  the  lessons  of  Fort  Bragg  is  the  importance  of 
orienting  the  staff  and  provider  network  to  the  values  and 
goals  of  the  demonstration  and  of  evaluating  staff  and 
providers  against  the  demonstration's  desired  outcomes. 
Because  differing  values  are  inevitable  among  staff  and 
providers,  it  is  essential  to  articulate  clearly  the 
demonstration's  values  as  a unifying  element  and  to  create 
an  in-house  process  to  air  and  resolve  differing  values. 
Without  such  a process,  staff  become  polarized  around 
different  philosophies,  instead  of  working  synergistically. 

Family  Involvement 

Perhaps  the  most  salient  lesson  about  family  involvement  is 
the  danger  in  assuming  that  everyone  means  the  same  thing 
by  "family  involvement, "-.the  importance  of  articulating 
clearly  what  the  demonstration  means  by  it  and  the 
importance  of  training  staff  in  the  demonstration's 
philosophy  and  service  approach  to  families.  There  needs  to 
be  clinical  leadership  to  help  families  make  good  decisions. 
Lack  of  strong  clinical  leadership,  particularly  at  a systems 
level,  affects  cost.  In  the  absence  of  such  leadership,  there  is 
a tendency  for  staff  and  providers  to  overload  families  with 
services  and  for  families  to  become  dependent,  aggravating 
lengths  of  stay  and  therefore  cost. 

Environmental  Context 

Systems  change  experiments  in  the  children's  arena  operate 
in  complex  political,  social,  and  interagency  environments. 
Particularly  in  the  children's  arena,  changes  in  one  system 
will  affect  all  other  child-serving  systems. 

Fort  Bragg  reinforces  the  usefulness  of  environmental  and 
ethnographic  analysis  for  anticipating  how  services  will  be 
used  and  for  informing  deliberations  about  optimal 
treatment  approaches  for  the  target  population.  For  example, 
it  might  be  anticipated  that  ready  access  would  be  a major 
factor  in  the  willingness  of  Army  families  to  use  services 
because  of  the  Army  mentality  of  "do  it  now."  Similarly,  the 
location  of  services  off-base  might  also  have  been 
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anticipated  to  increase  demand  for  services  because  of  the 
preference  of  military  families  to  use  non-base  providers, 
particularly  for  mental  health  services  where  stigma  is  an 
issue.  The  transiency  of  the  military  family,  particularly 
those  at  this  rapid  deployment  base,  might  have  argued  in 
the  design  phase  for  shorter-term,  problem-solving 
treatment  approaches  and  shorter  lengths  of  stay.  The  rapid 
deployment  nature  of  the  base,  which  has  the  effect  of 
depleting  medical  staff  from  the  base,  might  also  have  been 
anticipated  to  result  in  some  patient-shifting  from  the  Army 
medical  center  on-base  to  the  demonstration,  as  indeed 
occurred. 

Evaluation 

Just  as  it  is  essential  for  demonstration  implementors  to  be 
clear  about  their  purpose,  it  is  critical  that  evaluators  are 
careful  to  define  clearly  what  it  is  they  are  evaluating.  It  also 
is  critical  that  appropriate  caveats  be  included  regarding 
comparison  sites  that  are  used.  Typically,  there  are  political, 
operational,  and  other  constraints  that  influence  the  choice 
of  comparison  sites,  which  necessarily  affects 
comparability.  Those  involved  in  the  demonstration  believe 
that  the  most,  perhaps  only,  comparable  site  to  Fort  Bragg  is 
Fort  Hood,  which,  for  a number  of  reasons,  could  not  be 
used  (L.  Behar,  personal  communication.  Summer,  1995). 
Both  bases,  unlike  the  comparison  sites  that  were  used,  are 
involved  in  rapid  deployment,  house  specialized  forces, 
have  large  pediatric  populations,  have  high  birth  rates,  and 
have  a large  concentration  of  low  income  military 
personnel.  It  is  possible  that  these  factors  create  more 
difficult-to-treat  mental  health  problems.  If  this  is  the  case, 
then  the  fact  that  Fort  Bragg  had  outcomes  as  good  as,  and 
in  some  cases,  better  than  the  comparison  sites  that  were 
used,  is  encouraging.  Similarly,  a caveat  might  have  been 
noted  regarding  the  potential  influence  on  outcomes  of 
co-pays  and  deductibles,  with  which  families  at  the 
comparison  sites,  but  not  Fort  Bragg  families,  had  to 
contend.  Families  at  the  comparison  sites  may  have  been 
more  motivated,  due  to  the  presence  of  co-pays  and 
deductibles,  to  use  services  and  to  use  them  more 
efficiently.  This  motivation,  in  turn,  may  have  helped  to 
produce  better  treatment  outcomes. 

In  assessing  adequacy  of  implementation,  evaluators  need  to 
be  clear  about  the  stage  of  implementation  that  is  being 
evaluated,  as  systems  change  experiments  are  highly 
developmental  in  nature.  They  also  need  to  define  carefully 
what  aspects  of  implementation  are  being  assessed  and  the 
limitations  of  those  aspects  to  creating  a holistic  picture  of 
implementation.  The  Fort  Bragg  evaluation,  for  example, 
evaluated  case  management  and  intermediate  services  as 
indicators  of  adequacy  of  implementation.  The  evaluation 
did  not  assess  the  appropriateness  of  case  management  for 
the  various  populations  of  children  receiving  that  service, 
which  proved  to  be  an  issue.  The  evaluators  assessed 
intermediate  services  in  one  clump,  rather  than  separately. 
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making  it  impossible  to  determine  whether  certain 
components  were  more  or  less  well  developed  than  others, 
which  might  affect  results.  Also,  at  the  time  of  the 
implementation  evaluation,  the  demonstration  was  at  a 
relatively  early  stage  of  development,  besieged  by  a heavy 
demand  for  services,  an  influx  of  new  providers,  and 
without  well  developed  internal  management  and  control, 
quality  assurance  and  utilization  review  mechanisms,  all  of 
which  affected  results. 

Evaluators  need  to  be  very  careful  in  their  interpretation  of 
cost  findings.  In  Fort  Bragg,  one  might  have  concluded  that 
the  high  costs  associated  with  the  demonstration  were 
exactly  what  might  have  been  anticipated  given  its  design. 
This  is  a quite  different  interpretation  than  saying  that  costs 
were  higher  than  expected.  This  goes  back  to  the  earlier 
point  about  the  importance  of  both  the  demonstration  and 
the  evaluation  being  clear  about  purpose  and  goals. 

Summary 

Demonstrations  are  expected  to  undertake  many  new 
endeavors  very  quickly.  Fort  Bragg,  for  example,  had  to 
simultaneously  educate  the  community,  hire  new  staff, 
contract  with  outside  providers,  start  new  programs,  build 
internal  management  and  monitoring  mechanisms,  respond 
to  sponsor  demands  for  information  and  adjustments,  orient 
and  train  new  staff  and  providers,  find  facilities,  etc. 
Multi-faceted  systems,  changing  experiments  in  the 
children's  arena,  need  time  and  a deliberately  phased 
timetable,  with  benchmarks  along  the  way,  to  be 
implemented  (and  evaluated)  effectively. 
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Care:  Clinical  Outcomes  at  18  Months  for  the 
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Introduction 

In  the  face  of  the  seriously  fragmented  state  of  children's 
mental  health  services  in  the  United  States  (Knitzer,  1982), 
Stroul  and  Friedman  (1986)  proposed  that  children  receive 
services  in  a service  system  focused  around  a mental  health 
continuum  of  care  that  would  be  comprehensive, 
community  based,  family  focused,  and  culturally  competent, 
providing  services  in  the  least  restrictive  setting  possible. 
The  continuum  of  care  is  a full  range  of  mental  health 
services  that  includes  residential,  intermediate,  and 
nonresidential  services.  The  assumption  behind  the 
continuum  of  care  is  that  children  would  improve  more 
under  a continuum,  because  they  would  receive  more 
appropriate,  individuated  services  (Bickman,  1996a).  To 
date,  only  one  study,  the  Fort  Bragg  Evaluation  Project 
(FBEP;  Bickman  et  al.,  1995),  has  conducted  an  empirical 
test  of  this  assumption.  The  FBEP  found  no  differences  after 
one  year  in  clinical  outcomes  between  children  treated  in  the 
continuum  of  mental  health  services  of  the  Fort  Bragg  Child 
and  Adolescent  Mental  Health  Demonstration  Project  in 
Fayetteville,  North  Carolina  (the  Demonstration)!  and 
children  who  received  traditional  CHAMPUS2  services  at 
two  comparison  sites  (the  Comparison),  although  clients 
improved  at  both  sites.  This  led  the  evaluation  team  to 
conclude  that  the  continuum  of  care  theory  was  not 
supported. 

This  conclusion,  however,  has  been  disputed  (Behar,  1996; 
Friedman  & Bums,  1996;  Lane,  1996).  Although  these 
disputes  have  been  addressed  (Bickman,  1996b),  perhaps 
longer  term  follow-up  may  show  that  the  Demonstration 
was  more  effective  and  successful  for  children  with  SED 
(Friedman  & Bums,  1996).  This  paper  uses  data  collected 
by  the  FBEP  at  18  months  (Wave  4)  to  do  so. 

The  Demonstration  and  the  FBEP 
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Although  the  Demonstration  and  the  FBEP  have  been 
described  in  detail  elsewhere  (Bickman  et  al.,  1995),  they 
are  briefly  reviewed  here.  The  Demonstration  provided  a 
continuum  of  mental  health  services,  including  outpatient 
therapy,  day-treatment,  in-home  counseling,  therapeutic 
homes,  specialized  group  homes,  24-hour  crisis 
management  services,  and  acute  hospitalization.  Individual 
case  managers  and  interdisciplinary  treatment  teams  worked 
with  all  children  assigned  to  outpatient  therapy  in  order  to 
integrate  and  fit  services  to  the  needs  of  each  child. 
Treatment  plans  utilized  the  least  restrictive  service  options, 
and  services  were  community-based.  No  co-payments  were 
required  for  children  and  their  families. 

The  FBEP,  a quasi-experimental  study,  collected  mental 
health  outcome  data  on  984  children,  ages  five  to  17,  and 
their  families  to  evaluate  the  relative  effectiveness  of  the 
Demonstration.  Five  hundred  and  seventy-four  children 
received  mental  health  services  at  the  Demonstration  site. 
The  remainder  received  mental  health  services  at  two 
comparable  Army  installations.  Fort  Campbell  and  Fort 
Stewart.  Children  at  these  sites  received  CHAMPUS 
services.  Data  collection  occurred  within  30  days  of  entry 
into  the  service  system  (Wave  1),  six  months  (Wave  2),  12 
months  (Wave  3),  and  18  months  (Wave  4)  after  a 
participant's  entry  into  the  study.  The  data  includes 
structured  diagnostic  interviews,  behavioral  checklists, 
demographic  data,  and  other  relevant  backgroimd 
information.  The  children  and  adolescents  at  both  the 
Demonstration  and  Comparison  sites  were  comparable  at 
intake  on  background,  demographic,  and  initial  mental 
health  status  (Bickman,  et  al.,  1995).  Children  were 
predominantly  male  (63%),  white  (71%),  and  were  mostly 
from  middle  income  (54%  earned  between  $20,000  and 
$40,000)  two  parent  families  (80%)  in  which  at  least  one 
parent  had  some  higher  education  (87%  of  families  had  one 
parent  with  more  than  a high  school  education).  The  average 
age  was  1 1 years. 
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Methods 


Twelve  key  outcome  variables  for  Wave  4 (18  months)  were 
analyzed  using  a random  regression  (Gibbons  et  al.,  1993; 
Laird  & Ware,  1982)  or  a hierarchical  linear  model  (HLM; 
Bryk,  Raudenbush,  & Congdon,  1994;  Bryk  8c  Raudenbush, 
1992).  These  variables,  summarized  in  Table  1.  are  the  same 
key  variables  used  in  earlier  analyses  (Bickman  et  al., 

1995) ,  except  that  the  Global  Level  of  Functioning  (GLOF), 
an  adaptation  of  the  Child  Global  Assessment  Scale  (Shaffer 
et  al.,  1983),  was  dropped  because  it  is  redundant  with  the 
Child  and  Adolescent  Functional  Assessment  Scale 
(CAFAS:  Hodges,  1995),  and  the  Vanderbilt  Functioning 
Inventory  (VFI;  Bickman,  Lambert,  Summerfelt,  8c  Karver, 

1996) ,  a short  checklist  of  obvious  functioning  problems, 
was  added. 

Random  regression  or  HLM  overcome  problems  measuring 
change  inherent  in  classical  models,  such  as  measuring 
change  as  a two-wave  difference  score  versus  a covariance 
residual  or  deleting  whole  cases  missing  any  data  versus 
imputing  data  not  observed.  Random  regression  and  HLM 
use  all  observations  and  measure  time  exactly.  Subjects  are 
treated  in  these  analyses  as  a random  effect.  Groups  and 
explanatory  covariates  are  treated  as  fixed  effects.  If  the 
Demonstration  children  improved  more,  they  would  exhibit 
a significantly  steeper  slope  than  the  Comparison  children. 

Because  the  FBEP  had  a serious  amount  of  attrition  (i.e., 
completion  rates  over  4 waves  for  the  12  key  variables 
.ranged  from  65%  for  the  grand  total  pathology  measure  to 
81%  for  the  VFI),  and  because  attrition  rates  varied 
significantly  between  sites  for  several  measures  (i.e., 
participants  from  the  Demonstration  had  higher  attrition  on 
the  CBCL,  the  YSR,  and  the  VFI),  an  attrition  analysis  was 
performed  to  determine  whether  attrition  influenced 
differences  in  rates  of  clinical  change  by  site.  Two  random 
coefficients  outcomes  analyses  were  conducted  for  the  1 2 
key  variables  with  "missingness"  added  in.  "Missingness" 
was  first  defined  categorically  as  missing  either  one  or  more 
waves  or  missing  none,  and  then  as  the  number  of  missing 
waves.  If  "missingness"  influenced  results,  then  there  would 
be  significant  wave  by  site  "missingness"  effects. 

In  each  approach,  only  one  result  was  significant  (the  VFI  in 
the  unbalanced  analysis,  the  Most  Intense  Child  Reported 
Psychopathology  in  the  balanced  analysis;  p < 0.05). 
Because  only  one  of  12  results  was  significant,  and  the  two 
different  approaches  show  significant  results  for  different 
outcome  variables,  it  can  be  concluded  that  the  influence  of 
missing  data  on  the  outcome  analyses  was  negligible. 

Results 

The  belief  that  the  Demonstration  would  result  in  improved 
outcomes  for  children  over  time  is  not  bom  out  by  the 
mental  health  measures  at  1 8 months  for  the  entire  FBEP 
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mental  health  measures  at  1 8 months  for  the  entire  FBEP 
sample.  Only  two  of  the  12  key  variables,  the  CAFAS  and 
the  VFI  (both  measures  of  functioning)  showed  significant 
site  differences,  and  these  results  contradicted  each  other 
(see  Table  2).  According  to  the  CAFAS,  children  at  the 
Demonstration  site  were  functioning  better  at  1 8 months, 
while  according  to  the  VFI,  children  at  the  Comparison  site 
functioned  better. 

When  the  same  analyses  are  restricted  to  the  317  children 
with  serious  emotional  disorders  (children  with  any  PCAS 
diagnosis  at  intake  and  a GLOF  score  < 61),  the  picture 
changes  slightly  (see  Table  3).  The  same  contradictory 
results  are  found  for  the  CAFAS  and  the  VFI,  yet  the 
Demonstration  is  more  effective  according  to  the  YSR. 

Conclusions 

The  claims  that  services  in  the  Fort  Bragg  Demonstration's 
continuum  of  care  will  be  proven  more  effective  than  a 
traditional  fee-for-service  system  of  care  over  a longer 
period  of  time  and  that  the  Demonstration  was  effective  for 
children  with  SED  were  not  supported.  At  1 8 months,  only 
one  of  12  key  mental  health  outcome  variables  (CAFAS) 
favored  the  Demonstration  for  all  children  in  the  study,  and 
that  finding  was  contradicted  by  results  from  the  VFI  that 
favored  the  Comparison.  For  children  with  SED,  only  two 
outcomes  favored  the  Demonstration  (CAFAS  and  YSR) 
while  the  VFI  again  favored  the  Comparison.  Even  if  the 
results  for  the  VFI  are  discounted  because  its  reliability  and 
validity  have  not  yet  been  established,  the  Demonstration 
still  does  not  appear  substantively  more  effective.  Even 
when  looking  only  at  children  with  SED,  the  fact  that  only 
two  of  the  twelve  outcome  variables  favor  the 
Demonstration  at  1 8 months  suggests  that  the  continuum  of 
care  at  the  Demonstration  site  did  not  result  in  relative 
greater  improvement. 

It  is  clear  from  this  data  that  the  earlier  conclusions  from  the 
FBEP  are  supported  by  the  18  month  findings.  Children  at 
both  sites  improved  equally,  and  the  idea  that  a continuum 
of  care  yields  better  mental  health  outcomes  remains 
unsupported.  Also,  given  the  Demonstration  has  not  proven 
more  effective  over  the  18  months,  there  is  little  rationale  to 
expect  that  a Demonstration  effect  will  become  evident  even 
further  down  the  line.  In  light  of  these  findings,  researchers 
and  policy  makers  need  to  reconsider  the  continuum  of  care. 
If  a continuum  of  care  can  be  implemented  in  a more  cost 
efficient  manner  than  it  was  at  Fort  Bragg,  it  is  possible  that 
the  benefits  of  the  Demonstration  that  were  shown  (i.e., 
increased  access  to  more  coordinated,  individual,  and  less 
restrictive  care  community  based  services  without  a loss  in 
the  quality  of  care)  will  be  considered  adequate  to  justify 
implementing  continuums  of  care  for  children's  mental 
health  services  elsewhere. 
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Introduction 


In  August,  1991,  the  Youth  Incentive  Program  (YIP),  was 
officially  launched  in  New  Jersey.  The  program,  which 
represented  a major  restructuring  of  children's  mental  health 
services,  grew  out  of  earlier  private  and  public  efforts  and 
reflected  national  trends  in  the  reform  of  mental  health 
services  for  children.  Cutting  across  most  of  these  programs 
is  the  underlying  principle  that  home-based,  family-centered 
care  within  an  integrated  cross-system  interagency  structure 
will  best  serve  troubled  children  and  adolescents. 

As  initially  set  up,  the  YIP  consisted  of  an  administrative 
unit  and  service  team  in  each  of  the  21  counties  of  New 
Jersey.  The  administrative  structure,  the  County  Interagency 
Coordinating  Council  (CIACC),  was  created  for  the  purpose 
of  establishing  policies  and  structures  to  increase 
coordination  and  monitor  interagency  services.  The  service 
units,  called  the  Case  Assessment  Resource  Teams  (CARTs), 
were  multi-disciplinary  teams  charged  with  making  a 
comprehensive  assessment  of  the  target  youth  and  family  and 
the  development  and  follow-up  of  multi-system  treatment 
plans.  The  priority  target  population  was  children  and  youth 
with  special  emotional  needs  in  the  child  welfare,  juvenile 
justice  and  mental  health  systems,  who  were  at  risk  of 
institutional  care. 
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Method 


Two  years  after  its  inception,  a state  funded  process 
evaluation  was  undertaken  to  determine  the  degree  to  which 
the  program  was  achieving  its  key  goals.  The  evaluation 
utilized  data  gathered  through  surveys  as  well  as  statistics 
and  information  collected  for  administrative  purposes  by  the 
state.  Three  different  surveys  were  utilized:  (a)  a 
self-administered  questionnaire  completed  by  the  21  CART 
Coordinators  in  the  state;  (b)  a telephone  interview 
completed  by  97  out  of  1 07  sampled  CART  members  from 
the  27  state  CARTs;  and  (c)  in-person  interviews  with  38 
parents  randomly  selected  from  four  geographic  regions  of 
the  state,  and  28  of  their  adolescent  children. 

Six  major  research  questions  addressing  critical  aspects  of 
the  service  system  model  guided  the  evaluation: 

• To  what  extent  was  the  identified  target  population 
being  served? 

• To  what  extent  did  the  system  of  care  represent  a 
multi-discipline,  multi-system  approach  to  delivering 
comprehensive  integrated  services  to  youth  and  their 
families? 

• To  what  extent  did  CART  members,  parents,  and 
children  believe  that  the  CARTs  facilitated  the  active 
involvement  of  parents  and  youth  in  service  planning? 

• To  what  degree  did  parents  and  children  actually 
participate  in  service  planning? 

• To  what  extent  were  services  comprehensive, 
non-traditional  and  family-centered  as  well  as 
child-focused? 

• How  helpful  was  the  program  to  the  children  and  their 
families? 

Results 

The  children  who  were  served  (ranging  from  8 to  1 8 years  of 
age)  clearly  met  the  criteria  for  the  target  population.  They 
came  from  the  child  welfare,  juvenile  justice,  and/or  mental 
health  systems  and  were  at  risk  of  residential  placement. 
Nearly  all  the  children  (97.3%)  had  previously  been  placed 
out  of  the  home  or  were  hospitalized.  They  had  an  extensive 
history  of  serious  emotional  disorders.  Seventy-five  percent 
had  been  hospitalized  for  a psychiatric  illness.  The  mean  age 
of  first  placement  was  nearly  1 1 years  of  age.  On  average, 
the  child  was  eight  when  the  parent  first  became  aware  that 
the  child  had  a problem.  Over  80%  of  the  children  had  taken 
medication  for  an  emotional  problem. 

One  important  program  goal  was  to  provide  a structure  that 
was  multi-system  and  multi-discipline  to  provide  integrated 
services.  Our  indicators  showed  that,  indeed,  the  CART 
membership  represented  a broad  spectrum  of  disciplines 
from  diverse  community  providers.  CART  members  listed 
more  than  ten  areas  of  professional  training,  with  the 
following  four  disciplines  identified  by  over  80%  of  the 
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following  four  disciplines  identified  by  over  80%  of  the 
respondents:  social  work,  psychology,  educational 
counseling,  and  corrections.  All  CARTs  were  comprised  or 
representatives  of  three  community  service 
systems&endash;mental  health,  child  protective  services,  and 
youth  services  (shelter,  detention,  probation,  family  court, 
etc.).  Two-thirds  or  more  of  the  CART  teams  also  had 
client/parent  advocates,  children's  clinical  case  managers, 
representatives  of  the  educational  system  (usually  child  study 
team  staff),  and  professionals  from  a broad  range  of  other 
community  based  agencies  (e.g.,  companionship  programs, 
substance  abuse  programs,  and  programs  for  the  prevention 
of  child  abuse  and  neglect). 

These  providers  worked  well  together  and  CART  members 
rated  their  teams  in  terms  of  five  factors  pertinent  to  the 
quality  of  working  together:  1)  the  team  members'  training, 
experience  and  philosophy,  2)  members'  understanding  of  the 
CART  mission,  3)  members'  acceptance  of  the  CART 
mission,  4)  level  of  cooperation,  and  5)  equal  regard  for  the 
opinions  and  decisions  of  all  standing  members.  Percentages 
of  the  sample  rating  their  team  good,  very  good,  or  excellent 
on  these  dimensions  ranged  from  87%  to  95%. 

On  the  negative  side,  the  Team  membership  was  not 
representative  of  the  target  population  in  terms  of  ethnicity 
and  socioeconomic  status.  Three-quarters  of  the  sample  of 
CART  members  were  Caucasian;  17.5%  of  the  sample  were 
African-Americans,  and  3.1%  were  Hispanic.  The  balance 
represented  primarily  Asian  Americans.  This  is  in  marked 
contrast  to  the  target  population  which  is  made  up  of  high 
proportions  of  African-American  and  Hispanic  families 
(exact  figures  are  not  available).  Supporting  the  demographic 
data  are  the  responses  of  CART  members  to  the  item  of 
whether  the  CART  was  "representative  of  the  community"  in 
terms  of  race,  ethnicity,  and  socioeconomic  status.  Nearly 
half  (46%)  rated  the  CARTs  as  deficient  in  this  respect. 

With  respect  to  parent  and  youth  participation  in  the 
program,  the  CART  program  appeared  to  be  doing  well  in 
certain  respects  and  poorly  in  others.  CART  members  and 
parents  alike  reported  that  CARTs  reach  out  to  them.  The 
majority  of  parents  (63.2%)  were  involved  in  developing 
service  plans,  and  62.1%  of  the  parents  were  involved  in 
providing  input  regarding  needs,  issues,  and  concerns. 
However,  a substantial  minority  of  parents  (37.4%)  felt  that 
they  were  not  receiving  enough  support  and  encouragement 
from  CARTs,  and  35.2%  of  the  parents  felt  that  they  did  not 
have  enough  input  into  service  planning.  Thus  far,  parents 
did  not  appear  to  be  participating  as  integral  members  of  the 
Team,  and  the  more  traditional  professional-client 
boundaries  remained  intact.  These  results  suggest  that 
parents  need  to  be  empowered  to  a greater  degree. 

Although  CART  members  reported  that  youth  were 
encouraged  to  participate  in  planning  services,  most  youth 
reported  only  minimal  involvement.  Only  a quarter  of  the 
youth  said  they  had  been  involved  in  developing  the  CART 
plan  and  had  a part  in  carrying  out  the  plan.  Engaging  the 
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plan  and  had  a part  in  carrying  out  the  plan.  Engaging  the 
youth  in  the  CART  process  appears  to  require  more 
aggressive  efforts  on  the  part  of  CARTs. 

Smaller  Family  Child  Teams  are  being  established  in  certain 
counties.  With  the  widespread  use  of  such  Teams,  the 
CARTs  may  move  closer  to  their  goal  of  establishing 
professional-parent  partnerships. 

Families,  not  just  the  youth  themselves,  reported  receiving 
services  addressing  a broad  range  of  needs,  thus  providing 
evidence  that  the  services  are  family-based,  as  well  as 
youth-centered.  Services  were  classified  as  non-traditional 
(e.g.,  paying  for  appliance  repair),  quasi-traditional  (e.g., 
companion  program),  and  traditional  (e.g.,  case  management, 
psychological  counseling).  While  traditional  services  were 
the  standard,  received  by  nearly  all  families,  almost 
two-thirds  received  quasi-traditional  services  and  more  than 
half  received  non-traditional  services.  Ratings  of  helpfulness 
varied  little  by  category. 

The  most  critical  question  of  all  concerned  how  helpful  the 
new  program  was  to  the  children  and  their  families.  The 
present  data  are  based  on  our  interviews  with  the  38  parents. 
We  make  no  claim  to  have  definitive  answers  to  that 
question.  The  final  answers  must  await  the  results  of  a larger 
statewide  study  using  more  rigorous  methods.  However,  the 
data  do  provide  some  grounds  for  optimism. 

The  approach  followed  was  developed  by  Magura  & Moses 
(1986)  and  has  demonstrated  reliability  and  validity.  The 
parent  interview  schedule  contained  questions  about  four 
types  of  problems:  (a)  the  child's  school  adjustment,  (b) 
conduct  or  behavioral  problems,  (c)  psychological  problems, 
and  (d)  parenting  skills.  For  each  specific  set  of  problems, 
parents  were  first  asked  to  indicate  whether  the  problem 
existed  when  they  first  talked  with  someone  from  CART.  If 
an  initial  problem  was  noted,  they  were  asked,  "Is  this  still 
true?"  If  the  problem  did  not  exist  initially,  the  parents  were 
asked  "Is  this  true  now?" 

Questions  pertaining  to  specific  problems  were  followed  by 
more  general  questions  in  which  respondents  were  asked  to 
indicate  for  each  of  the  four  areas  the  extent  to  which  CART 
services  had  contributed  to  the  improvement. 

For  each  of  the  four  problem  areas  an  improvement  score 
was  obtained  by  calculating  the  percentage  of  initially 
identified  school  problems  on  which  positive  change  had 
occurred  by  the  time  of  the  interview.  Potential  score  ranges 
were  zero  to  1 00%.  Categories  were  established  based  on  the 
percentage  of  problems  identified  initially  that  were  resolved 
by  the  time  of  the  interview.  Considerable  improvement  was 
represented  by  scores  of  50%  and  higher  (i.e.,  at  least  half  the 
initial  problems  had  been  resolved);  some  improvement  by 
scores  greater  than  0%  to  less  than  50%  (i.e.,  less  than  50% 
of  the  problems  had  been  resolved);  and  no  improvement  by 
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scores  of  0%  (i.e.,  none  of  the  initially  identified 
school-related  problems  had  been  resolved).  The  results  are 
shown  in  Table  1.  By  and  large,  the  program  seemed  to 
correspond  with  positive  changes  for  the  youth  and  their 
families.  Outcomes,  as  well  as  other  findings  based  on  our 
interviews  with  the  small  sample  of  parents  and  their 
children  are  tentative  and  will  need  to  be  examined  in  a 
larger  state- wide  longitudinal  study. 

The  CART  program  appears  to  be  measuring  up  in  many 
respects  but  falling  short  in  some.  Top  priority  is  being  given 
to  children  with  special  emotional  needs  who  are  at  risk  of, 
or  referred  for,  residential  treatment.  Team  membership 
represents  a broad  range  of  clinical  and  case  planning 
expertise.  Greater  attention,  however,  needs  to  be  given  to 
the  composition  of  the  CARTS  so  that,  demographically, 
they  are  more  representative  of  the  target  families.  Steps 
need  to  be  taken  to  socialize  both  professionals  and  clients 
into  new  roles  which  bring  them  into  a more  equal 
partnership.  A limited  exploratory  study  of  families  provides 
grounds  for  optimism  about  the  program,  showing  that 
families  and  youth  received  the  intended  comprehensive  and 
non-traditional  services  and  that  they  appeared  to  be  helpful. 
Given  these  preliminary  results,  it  is  critical  that  a more 
rigorous  outcome  evaluation  with  a larger  sample  be 
launched,  and  that  the  program  incorporate  a formal  and 
ongoing  system  of  monitoring  both  process  and  outcome  on 
a statewide  basis. 
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Introduction 


The  Health  Care  Reform  Tracking  Project  is  a five-year 
study  designed  to  track  and  analyze  state  health  care  reform 
initiatives  as  they  affect  children  and  adolescents  with 
emotional/behavioral  disorders  and  their  families.  This  is 
the  only  national  study  tracking  state  health  care  reforms  as 
they  affect  this  population.  It  is  being  conducted  jointly  by 
the  Research  and  Training  Center  for  Children's  Mental 
Health  at  the  University  of  South  Florida,  the  Human 
Service  Collaborative  of  Washington,  D.C.  and  the  National 
Technical  Assistance  Center  for  Children's  Mental  Health  at 
Georgetown  University. 
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Method 

The  first  activity  of  this  project,  completed  in  May  1995, 
was  a baseline  survey  of  all  fifty  states  to  identify  and 
describe  current  state  reforms.  The  survey  was  sent  to  State 
Child  Mental  Health  Directors  and  State  Medicaid 
Directors.  Responses  were  received  from  all  50  states  and 
the  District  of  Columbia.  The  baseline  survey  will  be 
repeated  in  years  3 and  5 to  see  how  things  change  over 
time. 

The  second  phase  of  the  study  involves  an  in-depth  study  of 
reforms  in  a smaller  sample  of  states,  tracking  over  time  the 
effects  on  service  delivery  to  children  with  emotional 
disorders  and  on  systems  of  care  for  this  population. 
Structured  telephone  interviews  and  site  visits  will  be 
conducted  annually  for  this  part  of  the  project  and  will 
include  data  collection  from  a broad  range  of  key 
stockholders,  including  parents,  state  administrators,  local 
providers,  and  advocates.  Questions  explore  a broad  array  of 
access,  quality,  cost  and  other  issues. 

The  1995  State  Survey  Results 

The  vast  majority  of  states  (86%)  are  involved  in  some  type 
of  health  care  reform  activity.  Nearly  all  health  care  reforms 
are  focusing  on  Medicaid,  and  most  (84%)  involve 
Medicaid  waivers. 

Most  states  are  in  early  stages  of  implementation  or 
planning.  Most  reforms  involve  both  physical  health  and 
mental  health.  However,  nearly  a third  of  the  reforms  are 
still  limiting  mental  health  coverage. 

Of  the  reforms  reported  to  us,  most  include  substance  abuse 
services  as  well  as  mental  health.  However,  reforms 
focusing  on  mental  health  services  are  less  likely  to  include 
substance  abuse  services;  only  half  of  the  mental 
health-only  reforms  include  some  provisions  for  substance 
abuse  services.  This  points  out  a lack  of  coordination 
between  the  mental  health  and  substance  abuse 
communities,  which  may  be  planning  their  reforms 
independently.  Considering  the  high  co-morbidity  between 
mental  health  and  substance  abuse  problems,  this  approach 
ultimately  may  not  make  much  sense. 

Most  reforms  focus  on  the  entire  Medicaid  population 
(60%)  or  some  portion  of  the  Medicaid  population.  If  states 
are  incrementally  phasing  in  the  Medicaid  population,  our 
data  show  that  they  are  most  likely  to  start  with  the 
population  which  receives  Aid  to  Families  with  Dependent 
Children  (AFDC).  In  almost  all  cases,  the  reforms  cover 
both  adults  and  children  (88%).  Interestingly,  if  there  is  a 
special  age-based  focus,  this  tends  to  be  on  children  rather 
than  on  adults. 

In  about  72%  of  the  reforms,  representatives  of  the  state 
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In  about  72%  of  the  reforms,  representatives  of  the  state 
mental  health  agency  were  involved  in  the  planning  process. 
In  most  but  not  all  cases,  when  the  mental  health  agency 
was  involved,  staff  from  the  children's  mental  health  agency 
also  were  involved.  However,  when  the  reform  covered  both 
physical  health  and  mental  health,  in  about  40%  of  the 
reforms,  state  mental  health  agency  staff  were  not  at  all 
involved  (and,  of  course,  neither  were  children's 
representatives). 

With  respect  to  differential  coverage,  children  are  more 
likely  to  get  different  treatment,  which  generally  represents 
an  expanded  or  enriched  benefit  package.  However,  the 
reality  is  that  two-thirds  of  the  reforms  do  not  yet  recognize 
that  children  may  need  a different  service  array  or  an 
enriched  benefit  package. 

Most  states  are  using  some  type  of  carve  out  arrangement 
for  behavioral  health  services  (72%).  Some  states  have 
carved  out  mental  health  completely  and  are  financing  and 
administering  these  services  separately  from  physical 
health.  Others  have  divided  the  mental  health  benefit  (i.e., 
they  provide  partial  mental  health  coverage  with  the 
physical  health  benefit  and  then  organize  a separate  delivery 
system  for  persons  with  serious  disorders  or  leave  this 
population  out  of  their  managed  care  systems).  Still  others 
are  not  using  carve  outs,  implying  that  mental  health 
services  are  integrated  with  the  physical  health  service 
delivery  systems.  We  found  a great  deal  of  confusion  about 
the  term  "carve  out;"  the  term  was  used  in  some  cases  to 
describe  arrangements  whereby  certain  services  or 
population  groups  are  actually  left  out  of  the  managed  care 
system. 

The  vast  majority  of  the  reforms  (88%)  involve  the  use  of 
capitation.  All  of  the  mental  health-only  reforms  use 
capitation.  States  are  developing  separate  capitation  rates  for 
a number  of  distinct  populations  (e.g.,  for  children  in 
general,  children  with  serious  emotional  disorders,  and 
children  in  state  custody).  Most  states  are  using  costs 
associated  with  prior  utilization  as  the  basis  for  determining 
capitation  rates. 

In  most  states,  only  the  state  Medicaid  and  mental  health 
agencies  are  contributing  dollars  to  finance  the  capitation 
rate.  In  very  few  instances  did  states  report  that  other 
child-serving  systems  are  contributing  funds  to  finance 
capitation  rates  for  children  with  emotional  disorders. 

With  regard  to  risk,  more  than  half  of  the  reforms  are 
reported  to  be  using  some  type  of  risk  adjustment  (61%)  to 
protect  children  with  serious  disorders  and  to  protect 
providers.  States  typically  are  protecting  themselves  as 
much  as  possible  from  financial  risk. 

States  are  using  a variety  of  types  of  entities  as  managed 
care  organizations;  in  most  cases  states  are  using  multiple 
entities.  There  is  extensive  use  of  for-profit  entities,  but  a 
significant  portion  of  the  reforms  are  using  regional  or  local 
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significant  portion  of  the  reforms  are  using  regional  or  local 
mental  health  authorities  or  community  mental  health 
centers  as  the  managed  care  organization.  The  data  suggest 
that  when  the  reform  is  focusing  on  mental  health  only, 
states  are  more  likely  to  use  their  existing  mental  health 
structure  for  planning  and  administering  service  delivery 
than  if  it  is  a broader  reform.  The  reforms  covering  both 
physical  health  and  mental  health  seem  to  be  relying  more 
on  the  for-profit  managed  care  entities  or  the  for-profit 
managed  behavioral  healthcare  entities.  Less  than  half  of  the 
reforms  are  designating  "essential  providers"  or  providers 
that  managed  care  organizations  are  obligated  to  use. 

States  are  using  a wide  range  of  managed  care  techniques  in 
the  reforms  they  reported  to  us,  including:  (a)  screeners  or 
gatekeepers,  (b)  case  management,  (c)  precertification,  (d) 
concurrent  review,  (e)  utilization  review,  and  (f)  preferred  or 
exclusive  provider  arrangements.  Only  about  a third 
reported  that  their  reforms  involved  organized  systems  of 
care  for  children. 

About  half  of  the  reforms  report  having  some  special 
management  mechanisms  for  children  with  serious 
emotional  disorders.  These  include  mechanisms  like 
intensive  case  management,  provisions  for  service  planning 
by  interagency  teams,  and  access  to  an  enriched  benefit 
package  as  noted  earlier. 

Around  outcomes,  we  found  that  reforms  are  focusing  on  a 
range  of  dimensions.  Cost,  access,  and  utilization  are  the 
areas  looked  at  most  closely.  Client  outcomes  and  program 
effectiveness  are  the  dimensions  considered  least  in  the 
reforms.  Nearly  70%  of  the  reforms  reported  that  they  are 
going  to  look  at  parent  satisfaction  as  a dimension  of 
outcome  assessment. 

Impact  Analysis  Preliminary  Results 

In  the  second  phase  of  the  Health  Care  Reform  Tracking 
Project,  now  underway,  we  are  exploring  in  greater  depth 
the  impact  of  state  health  care  reforms  on  children  with 
emotional  disorders  and  their  families  and  on  systems  of 
care  for  this  population.  Preliminary  results  were  obtained 
from  structured  telephone  interviews  with  13  states,  which 
included:  Arizona,  California, 

Hawaii,  Iowa,  Massachusetts,  North  Carolina,  Oregon, 
Pennsylvania,  Rhode  Island,  Tennessee,  Utah,  Washington, 
and  Wisconsin.  These  states  were  selected  because  they 
have  some  measureable  experience  with  managed  care  and, 
as  a group,  provide  differences  in  approach  and  geographic 
diversity.  The  preliminary  information  that  follows  is 
described  primarily  to  give  a sense  of  the  kinds  of  issues 
being  explored  in  the  Phase  II  work.  In  most  states,  it 
simply  is  too  early  in  the  managed  care  initiative  process  to 
determine  the  impact  on  children  and  families.  Hence,  these 
issues  will  be  followed  more  closely  in  follow-up  visits. 
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The  vast  majority  of  states  report  that  it  is  too  early  to  tell 
what  the  impact  is  on  child  outcomes,  if  indeed  this  is  even 
being  tracked.  Some  states,  however,  are  beginning  to 
identify  trends  in  service  access. 

Three  of  the  ten  states  indicate  that  their  health  care  reforms 
have  made  it  easier  for  children  to  obtain  mental  health 
services,  and  two  of  the  ten  states  say  reform  has  made  it 
more  difficult,  largely  because  medical  necessity  criteria 
restricts  access  to  mental  health  services  for  children  and  the 
number  of  providers  is  more  limited.  Three  of  the  ten  states 
also  indicate  that  more  children  are  using  mental  health  and 
substance  abuse  services  as  a result  of  their  managed  care 
initiatives.  This  is  due  to  expanded  coverage  for  uninsured 
and  poor  children  and/or  more  Medicaid  children  using 
services.  The  other  states  say  that  it  is  still  either  too  early  to 
tell,  or  that  there  has  been  no  effect  on  utilization.  Most 
states  are  unable,  at  this  point,  to  answer  the  question  as  to 
whether  access  to  mental  health  and  substance  abuse 
services  for  different  subgroups  of  children  (i.e.,  children  of 
color,  children  in  child  welfare,  children  in  juvenile  justice, 
children  in  special  education,  children  with  serious 
emotional  disorders)  is  greater  or  less  as  a result  of  their 
managed  care  initiatives.  It  is  unclear  as  to  whether  this  is 
because  it  is  too  early  to  tell  or  whether  states  simply  are  not 
tracking  access  in  this  manner. 

A split  among  the  states  exists  over  whether  providers  are 
willing  to  participate  in  the  reformed  system.  There  is  also  a 
split  over  whether  the  health  care  reform  has  meant  a need 
for  more  or  fewer  specialized  children's  programs  and 
practitioners.  Six  states  report  new  types  of  providers  and 
programs  included  in  service  networks  through  their  health 
reforms,  including:  (a)  social  workers,  (b)  paraprofessionals, 
(c)  hospital  diversion  programs,  (d)  wraparound  services,  (e) 
in  home  services,  (f)  partial  hospitalization,  (g)  respite 
services,  and  (h)  school-based  services.  One  state  reported 
that  their  reform  reduced  the  number  of  providers  and 
programs. 

Most  states  report  that  more  types  of  home  and  community 
based  services  are  covered  through  their  health  care  reform 
than  previously.  Most  states  also  report  a reduction  in  both 
the  use  of  inpatient  psychiatric  hospitalization  for  children 
and  the  lengths  of  stay  while  utilizing  inpatient.  Two  states 
report  that  their  health  care  reforms  have  led  to  a reduction 
in  out-of-home  placement,  but  most  states  say  it  is  too  early 
to  tell.  One-third  of  the  states,  however,  report  an  increase  in 
the  use  of  residential  treatment  centers,  as  well  as 
"dumping"  of  children  into  residential  treatment  in  the 
public  sector. 

Three  states  report  increased  costs  associated  with  their 
health  care  reforms,  largely  due  to  increased  access  to 
services.  Two  states,  however,  report  no  effect  on  cost,  and 
half  say  it  is  still  too  early  to  tell.  Three  states  report  a 
change  in  expenditure  patterns,  with  a decrease  in  spending 
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change  in  expenditure  patterns,  with  a decrease  in  spending 
on  inpatient  and  an  increase  for  community-based  services. 
Most  states  do  not  know  the  impact  of  their  health  care 
reforms  on  costs  to  other  child-serving  systems,  such  as 
child  welfare.  Half  of  the  states  report  greater  interagency 
collaboration  as  a result  of  their  health  care  reforms,  but  two 
states  report  that  their  reforms  have  exacerbated  the  issue  of 
which  system  should  pay  for  which  services  for  which 
children.  Four  states  indicate  that  their  reforms  have 
enhanced  their  ability  to  pool  funds  across  children's 
systems,  but  most  states  say  it  is  too  early  to  tell  or  report  no 
effect.  Four  states  also  indicated  an  enhanced  ability  to  use 
Medicaid  to  finance  mental  health  services  for  children  as  a 
result  of  their  health  care  reforms;  one  state  reports  a 
diminished  ability. 

Half  the  states  report  that  the  health  care  reform  has 
facilitated  their  development  of  systems  of  care  for  children, 
and  half  say  it  is  too  early  to  tell.  Most  states  report  that 
their  reforms  have  also  facilitated  their  ability  to  provide 
flexible,  individualized  services  for  children,  but  that  it  is 
more  difficult  to  coordinate  these  services.  The  vast 
majority  of  states  report  that  the  role  of  their  public  child 
mental  health  systems  has  changed  as  a result  of  their  health 
care  reforms,  with  public  child  mental  health  assuming 
more  monitoring  and  training  responsibilities  and,  in  some 
cases,  becoming  strictly  a long  term  care  provider  of 
services.  Also,  most  states  report  more  integration  between 
the  public  and  private  sectors  as  a result  of  their  reforms. 

Four  states  report  an  expanded  role  for  primary  care 
providers  in  identifying  and  treating  mental  health  problems 
for  children  as  a result  of  their  health  care  reform,  and  some 
states  express  concern  over  both  the  need  for  better  linkages 
between  primary  care  and  mental  health  providers  and  the 
need  for  better  training  for  primary  care  providers  in  mental 
health  issues  in  children.  Most  states  do  not  know  or  report 
limited  impact  on  the  extent  of  family  involvement  as  a 
result  of  their  health  care  reforms.  Even  the  three  states  that 
reported  support  for  family  involvement  in  their  reforms 
indicated  a need  to  do  more. 

Most  states  find  it  too  early  to  tell  if  quality  of  services  has 
improved.  Two  states  report  better  quality,  and  one, 
diminished  quality,  largely  due  to  a reduced  provider 
capacity  and  restrictions  on  length  of  stay  that  is  leading  to 
increased  recidivism.  Most  states  also  say  it  is  too  early  to 
tell  or  they  do  not  know  the  impact  of  their  reforms  on  the 
cultural  competence  of  services  provided  to  children.  Three 
states,  however,  indicated  enhanced  cultural  competence 
due  to  inclusion  in  the  service  network  of  culturally 
competent  providers  and  training  in  this  area.  Finally,  most 
states  report  it  is  too  early  to  tell  if  their  reforms  facilitate  or 
make  worse  the  early  identification  of  mental  health 
problems;  however,  four  states  report  it  is  easier  to  intervene 
earlier  because  managed  care  moves  people  faster  through 
the  system. 
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States  listed  the  following  as  the  most  important  things  to 
avoid  in  health  care  reform  with  respect  to  children  with 
emotional  disorders  and  their  families: 

• Splitting  acute  and  long  term  care  responsibilities 
between  the  managed  care  entity  and  the  public 
sector; 

• cost  shifting  among  child-serving  systems  and 
between  the  managed  care  entity  and  children's 
systems; 

• exclusion  of  families  and  advocates  from  planning 
and  implementation  of  the  health  care  reform; 

• assuming  that  private  managed  care  organizations 
understand  public  sector  clients; 

• categorical  funding  of  the  health  care  reform; 

• lack  of  outcome  measures  for  children; 

• inadequate  provider  network  (i.e.,  not  properly 
trained,  not  sufficient  capacity,  or  not  right  types  of 
providers  in  network); 

• poor  understanding  of  prior  costs  and  utilization;  and 

• hurried  planning  and  implementation. 

Conversely,  states  recommended  that  the  most  important 
elements  to  include  in  state  health  care  reforms  are: 

• A single  system  for  children  with  serious  and  mild 
disorders,  for  acute  and  extended  care  needs; 

• a broad,  flexible  array  of  covered  services  in  the 
benefit  design; 

• a system  of  care  approach  for  children  with  serious 
disorders; 

• family  involvement  in  planning  and  implementation; 

• extensive  interagency  involvement  in  planning  the 
system; 

• mandate  interdisciplinary  treatment  teams  and 
interagency  linkages  in  contract  language; 

• delineate  funding  responsibilities  across  children's 
agencies; 

• create  a citizens  review  board  and  a complaints 
resolution  process; 

• have  good  data  systems  to  plan  and  monitor  the 

system;  j 

• undertake  extensive  public  education  about  the  new  I 

system;  and  j 

• go  slow&endash;create  a deliberate  and  open 
planning  process. 

I 

i 

i 

Discussion 

I 

Not  only  are  states  proceeding  quickly  with  their  reforms,  | 

but  they  are  moving  ahead  with  statewide  implementation 
for  the  most  part,  without  the  benefit  of  pilots  or 
demonstrations.  This  highlights  the  need  for  careful 
monitoring  and  evaluation  of  these  reforms  as  they  proceed.  | 
It  also  increases  the  likelihood  that  states  will  have  to  make  I 
corrections  to  their  systems  over  time. 
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The  majority  of  the  reforms  still  do  not  recognize  that 
children  may  require  a different  service  array,  an  expanded 
benefit  package,  or  special  requirements  for  service  delivery 
like  interagency  service  planning  or  family  involvement, 
creating  concern  about  the  limited  use  of  organized  systems 
of  care  for  children  with  serious  emotional  disorders  in 
managed  care  systems. 

The  use  of  carve  outs,  and  the  type  of  carve  out,  for 
behavioral  health  services,  is  an  area  in  which  states' 
approaches  are  evolving  and  are  likely  to  change  over  time. 
We  will  need  to  look  very  closely  at  how  well  states  and 
managed  care  organizations  are  able  to  manage  the 
boundaries  between  carved  out,  left  out,  and  integrated 
populations.  Otherwise,  we  will  have  a great  deal  of  cost 
shifting  and  fragmentation  of  service  delivery,  especially  for 
children. 

There  is  a need  for  better  data  to  use  as  the  basis  to 
determining  capitation  rates  for  children  with  emotional 
disorders.  A lot  more  work  is  needed  to  figure  out  how  to 
track  outcomes  for  children  and  families  in  managed  care. 
Much  of  the  ongoing  work  in  this  area  appears  to  be 
adult-focused. 

The  entire  issue  of  family  and  youth  involvement  needs 
further  exploration.  Most  states  reported  some  level  of 
family  involvement  in  planning  their  reforms,  but  family 
involvement  seems  peripheral,  for  the  most  part. 

Over  the  past  decade,  there  has  been  increasing  awareness 
of  the  differences  between  children's  and  adults'  services. 
These  and  other  differences  must  be  accounted  for  as  states 
develop  and  refine  benefits,  gatekeeping  systems,  treatment 
planning  and  review  systems,  quality  and  outcome 
measures,  and  all  other  facets  of  managed  care.  Given  the 
profound  implications  that  managed  care  is  likely  to  have, 
we  cannot  underestimate  the  importance  of  involving  those 
with  children's  expertise  (i.e.,  staff  and  families)  at  the 
earliest  stages.  There  is  also  a clear  need  to  better  integrate 
other  children's  systems,  such  as  child  welfare,  education, 
and  juvenile  justice,  into  the  planning  of  managed  care 
systems. 
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Introduction 


Beginning  in  January  1994,  North  Carolina  implemented  a 
program  that  blended  capitated  financing  with  public-sector 
managed  care  for  mental  health  and  substance  abuse 
services.  The  program  is  called  Carolina  Alternatives  (CA), 
and  it  covers  children  eligible  for  Medicaid.  This 
symposium  is  designed  to  outline  the  structure  of  CA  and 
present  stakeholders'  perspectives  on  the  first  two  years  of 
implementation.  The  stakeholders  to  be  represented  include 
(a)  the  State's  Division  of  Mental  Health,  Developmental 
Disabilities,  and  Substance  Abuse  Services  (MH/DD/SAS) 
and  its  local  branches  (Area  Programs);  (b)  the  state, 
private,  and  general  hospitals  that  provide  mental  health  and 
substance  abuse  services  to  children;  (c)  the  county  Social 
Service  Departments  (DSS);  and  (d)  both  the  children  who 
receive  services  and  their  parents.  Initial  cost  findings  on 
both  inpatient  and  outpatient  service  delivery  will  also  be 
presented. 
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Stakeholders'  Perspectives:  Overview  of 
Carolina  Alternatives 

Dan  Tweed,  Ph.D. 

During  the  late  1980's,  the  Health  Care  Finance 
Administration  (HCFA)  was  concerned  that  Medicaid  funds 
were  being  used  to  hospitalize  children  who  might  be  better 
treated  in  other  settings.  Medicaid  regulations  require  three 
criteria  before  a hospital  admission  qualifies  for  Medicaid 
reimbursement;  (a)  the  child  has  a mental  health  condition 
warranting  care  under  the  supervision  of  a physician;  (b)  the 
child  can  benefit  from  hospitalization;  and  (c)  more 
appropriate  services  are  not  available  in  the  child's 
community. 

In  North  Carolina,  HCFA's  concerns  led  to  the 
implementation  of  a Utilization  Review  (UR)  program  in 
August  of  1990.  The  program  was  designed  by  North 
Carolina's  Division  of  Medical  Assistance  (DMA)  and 
implemented  by  a private-sector  managed  care  company 
operating  under  contract  with  DMA.  Designed  to  ensure 
that  Medicaid  regulations  were  being  observed,  the  UR 
program  incorporated  two  of  the  fundamental  tools  of  a 
managed  care  system&endash;a  pre-certification  review 
process  designed  to  divert  children  from  hospitalization 
when  more  appropriate  forms  of  care  were  present,  and  a 
continued  stay  review  process  designed  to  ensure  that 
children  were  not  hospitalized  longer  than  clinically 
necessary. 

Only  a subset  of  North  Carolina's  hospitals  were  initially 
targeted  by  the  UR  program,  but  this  subset  accounted  for 
over  90%  of  Medicaid-reimbursed  inpatient  stays.  Analyses 
suggested  that  the  UR  program  appeared  to  be  effective. 
Pre-post  comparisons  suggest  significant  reductions  in  the 
number  of  inpatient  days  provided,  reductions  in  inpatient 
expenditures,  and  substantial  savings  for  both  North 
Carolina  and  the  Federal  Government. 

Effective  as  it  was,  however,  the  program  had  obvious 
limitations.  First,  the  program  was  designed  as  a simple  UR 
strategy.  Even  though  the  program's  mission  included 
determining  if  more  appropriate  community-based  services 
were  present,  it  had  no  care  management  component.  If  a 
child  was  denied  admission,  there  was  no  process  in  place  to 
ensure  that  the  child  received  an  appropriate  alternative 
service.  Second,  if  the  program  was  working  as  designed, 
the  potential  for  further  reductions  in  inpatient  utilization 
was  limited  by  the  lack  of  community-based  alternatives  in 
the  child's  community  of  origin.  Availability  of 
community-based  alternatives,  in  turn,  was  limited  by  the 
lack  of  funding  to  build  good  systems  of  care.  What  was 
needed  was  a managed  care  strategy  capable  of  generating 
savings  and  converting  those  savings  into  alternative 
community-based  services.  Under  the  UR  program,  savings 
stayed  in  state  and  federal  coffers. 
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stayed  in  state  and  federal  coffers. 

Carolina  Alternatives 

Beginning  in  April  of  1991,  North  Carolina's  Division  of 
MH/DD/SAS  collaborated  with  DMA  to  design  a new 
program,  Carolina  Alternatives  (CA),  which  could 
overcome  the  weakness  of  the  UR  program.  During  the  first 
two  years,  CA  was  implemented  in  10  of  North  Carolina's 
41  Area  Programs.  This  pilot  period  served  to  road  test  CA 
and  set  parameters  needed  for  statewide  implementation. 
Under  CA,  several  things  happened.  First,  the  local  Area 
Programs  became  responsible  for  the  management  of  all 
medicaid-reimbursed  services  delivered  to  children  in  their 
service  areas.  This  included  responsibility  for  the  utilization 
review  process  and  responsibility  for  finding  appropriate 
care  when  inpatient  care  was  deemed  inappropriate. 

Second,  in  order  to  carry  out  the  managed-care  role,  pilot 
programs  were  given  a prospectively  determined  budget 
with  which  to  provide  or  purchase  appropriate  forms  of 
care.  Pilot  programs  were  expected  to  live  within  this 
budget  and  were  at  full  risk  for  expenditures  in  excess  of  the 
budgeted  amount.  For  the  first  two  years  of  implementation, 
however,  the  pilot  programs  received  a prospectively 
determined  budget  having  two  distinct  components,  one 
expressly  designated  for  the  purchase  of  inpatient  services 
and  the  other  for  the  purchase  of  outpatient  services 
(including  the  full  range  of  non-inpatient  forms  of  care). 
Area  programs  were  at  full  risk  for  the  inpatient  component 
only.  Any  savings  accrued  from  diverting  children  to  less 
costly  community-based  alternatives  was  kept  by  the  Area 
Programs. 

By  contrast,  the  pilot  programs  enjoyed  a two  year' 
"moratorium"  on  risk  for  outpatient  expenditures.  Pilot 
programs  were  reimbursed  for  excess  outpatient 
expenditures,  while  unspent  money  was  returned  to  the 
state.  Outpatient  expenditures  during  the  moratorium  period 
were  used  to  determine  the  full  capitation  rate  that  went  into 
effect  at  the  end  of  the  moratorium  period.  Thus,  CA  created 
a strong  incentive  to  both  divert  children  from  inpatient 
services  and  spend  money  on  the  outpatient  side.  The  intent 
behind  this  incentive  structure  was  to  provide  a stimulus  for 
the  development  of  new  community-based  services. 

Carolina  Alternatives'  design  stimulated  the  development  of 
community-based  services  along  three  lines.  First,  savings 
from  reductions  in  the  number  of  inpatient  days  purchased 
were  used  as  flexible  dollars  to  buy  services  which  might 
otherwise  not  be  covered  under  NC's  mental  health  benefits 
structure.  Alternative  forms  of  residential  treatment  were 
included  here.  Second,  the  two  year  moratorium  on 
outpatient  risk  created  an  incentive  to  shift  the  costs  of 
caring  for  the  needs  of  children  with  serious  problems  to  the 
outpatient  side,  leading  Area  Programs  to  either  invest  in  the 
development  of  new  in-house  capacity  or  to  incorporate  an 
expanding  base  of  private  contract  providers.  Finally,  the 
program  provided  an  incentive  for  hospitals  to  offer 
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program  provided  an  incentive  for  hospitals  to  offer 
outpatient  alternatives  to  inpatient  care.  Faced  with  fewer 
admissions,  fewer  days,  and  fewer  dollars,  hospitals  were 
induced  to  enter  the  outpatient  services  market  in  a more 
aggressive  marmer. 

With  these  thoughts  in  mind  we  now  review  how  the 
program  has  been  working.  We  approach  this  question  in 
two  ways.  First,  we  share  the  views  of  several  key 
stakeholder  groups  whose  views  can  fundamentally 
condition  the  success  of  a program  like  CA:  (a)  the  pilot 
programs  implementing  CA  in  quite  varied  circumstances; 
(b)  the  inpatient  providers  whose  activities  have  been 
profoundly  affected  by  CA;  (c)  the  DSS  agencies  who  often 
see  the  same  clients,  compete  for  the  same  residential 
capacity  and  often  seek  mental-health  care  for  their  clients; 
(d)  the  patients  and  their  families;  and  (e)  the  state  Mental 
health  office.  Second,  we  review  expenditure  patterns  on 
inpatient  and  outpatient  services  to  determine  the  presence 
of  program  effects  on  expenditures. 

Stakeholders'  Perspectives:  Area  Programs, 
Hospitals  and  Departments  of  Social  Services 

Dalene  Stangl,  Ph.D. 

Introduction 

This  section  presents  the  perspective  of  three  groups 
regarding  how  CA  restructured  service  delivery, 
implemented  care  management,  and  redefined  interagency 
relations.  Problems  encountered  by  each  group  are  also 
presented.  The  three  groups  are  (a)  the  10  Area  Programs 
responsible  for  care  management,  (b)  the  hospitals  that 
provide  mental  health  and  substance  abuse  services  to 
children,  and  (c)  the  DSS  directors  for  each  county.  Each  of 
these  groups  has  an  important  stake  in  the  delivery  of 
mental  health  and  substance  abuse  services  to  children,  and 
hence,  each  group  provides  an  essential  perspective  to 
understanding  the  inner-workings  of  this  public-sector 
managed  care  program. 

Methods 

In  July  1994,  six  months  after  initial  implementation  of  CA, 
mail-out  questionnaires  designed  by  two  authors  (DT  and 
DS)  were  sent  to  all  41  Area  Programs.  These 
questionnaires  tapped  general  attitudes  toward  managed 
care  and  familiarity  with  CA.  In  addition,  the  10 
participating  Area  Programs  were  sent  a second 
questionnaire  that  asked  about  initial  management 
experiences.  These  latter  questionnaires  helped  prepare  the 
researchers  for  subsequent  interviews.  Personnel  from 
participating  Area  Programs  (i.e.,  CA  coordinators, 
utilization  review  managers,  management  information 
system  personnel  and  finance  personnel)  were  interviewed 
face-to-face,  on-site,  at  two  time  points.  The  first  interview 
was  conducted  by  two  of  the  authors  (DS  and  DT),  during 
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was  conducted  by  two  of  the  authors  (DS  and  DT),  during 
August/September,  1994.  These  interviews  assessed  the 
managed  care  strategy  and  service  structure  of  each  Area 
Program.  The  second  interview  occurred  during  February, 
1996,  26  months  into  the  program,  and  was  conducted  by 
one  of  the  authors  (DS).  It  was  preceded  by  a single 
mail-out  questionnaire  to  all  41  Area  Programs  that  asked 
updated  but  parallel  questions  to  the  previous 
questionnaires. 

Thirty-five  hospital  administrators  responsible  for 
negotiating  service  contracts  with  the  Area  Programs  were 
interviewed  over  the  telephone  at  two  points  in  time.  The 
first  interview  was  in  July/ August,  1994,  and  the  second 
interview  was  during  July/August,  1995.  The  first  of  these 
interviews  was  conducted  by  the  author  (DS)  and  the  second 
by  a research  assistant  trained  by  the  author  (DS). 

Finally  100  DSS  directors  were  interviewed  over  the 
telephone  during  July,  1995.  These  interviews  were 
conducted  by  a research  assistant.  Copies  of  interviews 
and/or  questionnaires  may  be  obtained  from  the  author 
(DS). 

Results 

Area  Programs 

Modeling  the  structure  of  CA  as  a wheel,  Medicaid  and  the 
State  Office  of  MH/DD/S AS  are  the  hub  and  the  Area 
Programs  are  the  spokes.  The  Area  Programs  carry  the 
weight  of  the  program  in  that  they  are  responsible  for 
implementation  and  are  at-risk  for  excess  expenditures 
above  and  beyond  the  capitation  amount.  They  have 
expended  a great  deal  of  time,  thought,  energy,  and  patience 
in  designing,  implementing,  and  adapting  to  an  evolving 
process.  Most  results  presented  here  will  be  from  the 
26-month  interviews  conducted  and  questionnaires  collected 
in  February  of  1996. 

Service  Delivery:  All  1 0 Area  Programs  reported  that  CA 
resulted  in  decreases  in  the  average  length  of  time  until 
children  received  first  treatment.  Estimates  ranged  from 
decreases  of  1 to  30  days,  with  5 days  representing  the 
median  decrease.  Of  the  1 0 Area  Programs,  9 reported 
increases  in  in-house  direct  care  staff  and/or  treatment  slots, 
in  residential  treatment  slots,  and  in  case  management  staff. 
All  10  Area  Programs  developed  and  maintained  an 
extensive  network  of  private  contract  providers.  The  number 
of  contracts  with  individual  care  providers  ranged  from  1 0 
to  78  (median  = 32),  and  with  group  providers,  ranged  from 
10  to  33  (median  = 12). 

Admission  rates  to  hospitals  from  the  1 0 Area  Programs 
decreased,  and  the  average  length  of  hospital  stays  were 
reduced  to  less  than  30  days,  with  9 of  10  Area  Programs 
reporting  average  lengths-of-stay  less  than  14  days.  Formal 
grievances  from  the  hospitals  were  minimal  with  most  Area 
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grievances  from  the  hospitals  were  minimal  with  most  Area 
Programs  experiencing  no  grievances  that  were  unresolved 
at  the  local  level. 

Care  Management:  Area  Programs  were  asked  to  report 
how  extensive  efforts  to  manage  inpatient  care  were  during 
the  first  two  years  of  the  program.  On  a scale  of  1 to  7,  (1  = 
Not  at  all  Intense,  4 = Moderately  Intense,  and  7 = 

Extremely  Intense),  eight  Area  Programs  rated  their  efforts 
as  5 or  higher,  with  4 rating  their  efforts  as  extremely 
intense.  The  2 remaining  Area  Programs  rated  their  efforts 
at  level  3.  On  the  outpatient  side.  Area  Programs  anticipate 
that  efforts  to  manage  outpatient  care  will  have  to  be  equally 
intensive  with  the  onset  of  outpatient  capitation. 

After  two  full  years  of  implementation,  only  3 of  the  10 
Area  Programs  felt  they  were  still  at  moderate  risk  of  deficit 
spending  for  inpatient  care.  The  remaining  7 reported  very 
little  or  no  risk  of  deficit  spending  for  inpatient  care.  The 
opposite  was  true  for  deficit  spending  on  the  outpatient  side. 
Here  7 Area  Programs  reported  being  at  moderate  to 
substantial  risk  of  deficit  spending.  The  other  3 programs 
reported  little  to  some  risk  of  deficit  spending. 

The  most  common  barriers  to  managing  inpatient  care 
reported  by  the  Area  Programs  was  the  lack  of 
community-based  alternative  services.  This  barrier  was 
reported  by  8 of  10  Area  Programs.  Other  barriers  reported 
by  at  least  3 Area  Programs  were  lack  of  experience  with 
the  managed  care  process  and  lack  of  clarity  from  the  state. 

The  two  most  commonly  reported  barriers  to  managing 
outpatient  care  were  poor  communication  with  service 
providers  and  fluctuating  expectations  as  the  Area  Programs 
move  from  outpatient  growth  to  containment.  Until  January, 
1996,  Area  Programs  were  provided  full  reimbursement  for 
outpatient  services.  Now  outpatient  services  are  also 
reimbursed  on  a capitated  basis.  Hence,  the  first  two  years 
of  CA  resulted  in  outpatient  service  expansion,  while 
January,  1996  marked  the  beginning  of  service  containment. 
Area  Programs  reported  nervousness  about  their  abilities  to 
maintain  service  provision  at  the  same  levels  as  the  first  two 
years  of  the  program.  Other  barriers  reported  by  at  least  3 
Area  Programs  included  lack  of  sufficient  community-based 
services,  conflicting  philosophies  with  service  providers, 
and  lack  of  clarity  from  the  state. 

Interagency  Relations:  As  gatekeepers  to  service  delivery 
and  payments.  Area  Programs  are  the  principle  decision 
makers,  and  this  may  present  a source  of  conflict  for  other 
agencies.  Competent  communication  and  negotiation  skills 
are  crucial.  Questions  pertaining  to  Area  Programs' 
relationships  with  Department  of  Social  Service,  Juvenile 
Justice,  and  Education  showed  that  while  most  relationships 
have  some  problems  and  some  fluctuated  frequently,  40% 
of  the  relationships  remained  the  same,  and  50%  improved 
since  the  beginning  of  CA. 

Other  Results:  Nine  of  10  Area  Programs  described 
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Other  Results:  Nine  of  10  Area  Programs  described 
dissatisfaction  with  their  management  information  systems 
in  at  least  one  of  the  following  areas;  claims  management, 
patient  tracking,  utilization  review,  and  patient  scheduling. 
All  Area  Programs  attained  savings  from  inpatient 
capitation  reimbursements  during  the  first  two  years  of  CA. 
Area  Programs  estimated  that  22  to  58%  of  that  savings  was 
spent  on  administrating  the  program. 

The  final  question  asked  of  Area  Programs  was:  "If  CA 
ended  today,  how  would  you  evaluate  what  it  has  done  for 
children's  mental  health  in  your  area?"  The  responses 
resonated  a common  chorus.  Area  Programs  reported  that 
CA  has  developed  greater  continuity  of  care  for  children  by 
the  improving  quality  and  quantity  of  services  and  requiring 
accountability  for  the  entire  spectrum  of  care.  Access  to 
services  has  improved,  with  more  children  being  served  in 
each  Area  Program.  Mental  health  centers  are  working 
pro-actively  with  communities  to  develop  a wide  spectrum 
of  services.  Communication  has  improved  so  that  now 
information  is  passed  with  the  child  across  episodes  and 
providers.  Accountability  has  increased,  both  in  terms  of 
fiscal  responsibility  as  well  as  service  provision.  CA  has 
provided  better  coordination  and  communication  between 
all  stakeholders  in  the  process. 

Hospitals 

Hospitals  are  an  important  stakeholder  in  CA.  As  service 
delivery  is  restructured  and  less  costly  outpatient  services 
substituted  for  inpatient  ones,  hospitals  stand  to  lose  the 
most.  Because  only  about  30%  of  children  eligible  for 
Medicaid  reside  in  the  vicinity  of  participating  Area 
Programs,  the  full  impact  of  CA  on  hospitals  has  yet  to  be 
seen,  but  change  is  evident.  As  of  July,  1995,  74%  of  the  35 
hospitals  serving  children  with  mental  health  and  substance 
abuse  problems  had  signed  a CA  contract  with  at  least  one 
Area  Program.  The  results  provided  below  are  from  the 
July,  1995  interviews  with  administrators  from  these 
hospitals. 

Service  Delivery:  Fourteen  percent  of  hospitals  reported 
plans  to  decrease  inpatient  capacity,  and  80%  of  those 
planning  a decrease,  attribute  their  action  to  CA.  Forty 
percent  report  plans  to  increase  their  outpatient  capacity, 
and  of  those  planning  an  increase,  43%  attribute  the  change 
to  CA. 

Care  Management:  As  of  July,  1995,  hospitals  had  signed  a 
total  of  61  contracts  with  the  10  Area  Programs.  Each 
hospital  rated  each  Area  Program  with  whom  they  had  a 
contract  on  several  dimensions  of  care  management. 
Hospitals  were  asked  whether  Area  Programs  exercised  too 
little,  about  right,  or  too  much  control  on  the  hospitalization 
process.  Sixty  percent  of  the  ratings  fell  in  the  about  right 
category,  and  27%  fell  in  the  too  much  category,  down  from 
44%  in  1994. 
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Hospitals  also  rated  Area  Programs  on  their  ability  to 
manage  inpatient  care.  Seventy-five  percent  of  the  ratings 
fell  in  categories  ranging  from  satisfactory  to  very  well. 
When  asked  to  compare  the  utilization  review  of  the  Area 
Programs  to  that  of  the  private  company  providing  the 
review  prior  to  CA,  35%  reported  that  the  Area  Programs 
were  more  responsive  to  the  needs  of  children;  22% 
reported  the  Area  Program  and  private  company  were  about 
equally  responsive;  22%  reported  Area  programs  were 
asked  about  responsiveness,  and  22%  did  not  know.  Similar 
percentages  were  reported  when  the  same  question  was 
asked  with  respect  to  the  needs  of  the  hospital  rather  than 
needs  of  the  children. 

Other  results:  Finally  hospitals  were  asked  to  rate  the 
impact  of  CA  on  the  children  they  served.  Forty-three 
percent  reported  that  CA  had  a favorable  or  extremely 
favorable  impact;  26%  reported  negligible,  and  23% 
reported  unfavorable. 

Departments  of  Social  Service 

In  most  Area  Programs,  the  transition  of  DSS  into  CA  has 
not  been  smooth.  Notions  of  medical  necessity  and 
treatment  versus  placement  have  been  slowly  accepted  by 
DSS  departments.  Area  Programs  worry  that  the  advances 
made  with  DSS  departments  during  the  growth  in  the 
uncapped  outpatient  services  period  will  be  lost  as  more 
stringent  efforts  at  managing  outpatient  services  are 
necessary  under  the  second  phase  of  CA.  Results  presented 
here  are  from  telephone  interviews  conducted  in  July,  1995. 

Service  Delivery:  DSS  directors  were  asked  the  following 
question:  As  a consequence  of  CA,  do  you  think  that  the 
mental  health  service  options  available  to  children  eligible 
for  Medicaid  (1)  Improved  very  much,  (2)  Improved,  (3) 
Remained  the  same,  (4)  Deteriorated,  (5)  Deteriorated  very 
much  or  (6)  Don't  know?  Ten  percent  of  the  DSS 
departments  in  CA  participating  counties  reported  some 
level  of  deterioration,  while  53%  reported  some  level  of 
improvement.  The  remaining  36%  fell  in  the  remained  the 
same  or  don't  know  categories. 

Care  Management:  DSS  directors  were  asked  the  following 
question:  Under  CA,  your  local  mental  health  agency  serves 
as  the  single  entry  point  for  children  needing  mental  health 
services;  given  your  experience  with  the  local  mental  health 
agency,  how  effective  do  you  think  they  have  been  in  this 
role?  Of  CA  participating  counties,  68%  responded  that 
Area  Programs  had  been  moderately  or  very  effective  in 
their  role.  Of  the  nonparticipating  counties,  who  will 
eventually  participate,  85%  expect  the  Area  Programs  will 
be  moderately  or  very  effective  in  their  managed  care  role. 

Interagency  Relations:  Directors  were  asked  to  describe 
problems  they  encountered  in  their  relationship  with  the 
Area  Programs.  The  most  frequent  problem  mentioned  by 
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Area  Programs.  The  most  frequent  problem  mentioned  by 
DSS  was  too  slow  or  infrequent  contact  between  agencies. 
This  was  true  regardless  of  whether  or  not  the  county  was 
currently  participating  in  CA.  Of  the  participating  counties 
(n  = 31),  23%  reported  this  problem,  while  36%  of 
nonparticipating  counties  (n  = 69)  reported  this  problem.  No 
other  problem  was  reported  by  more  than  10%  of  the 
participating  counties.  Among  the  nonparticipating  counties, 
more  than  10%  reported  problems  with  Area  Programs  in 
the  areas  of  untrained  staff,  understafFing,  misunderstood 
DSS  functions,  and  inability  to  serve  children  with  special 
needs.  These  percentages  ranged  from  12%  to  17%  of  the 
nonparticipating  counties. 

Conclusions 

CA  has  had  its  share  of  hurdles.  These  hurdles  are  reflected 
in  the  three  perspectives  presented  here.  The  coming  years 
will  be  no  different,  and  tensions  are  likely  to  be 
exacerbated  by  capitated  reimbursement  for  outpatient 
services  and  by  implementation  in  the  remaining  3 1 Area 
Programs.  Area  Programs  will  have  a tough  managed-care 
role  as  they  tighten  control  on  outpatient  services. 
Interagency  relations  will  continue  to  have  ups  and  downs. 
Hospitals  will  continue  to  lose  demand  for  inpatient  services 
and  need  to  substitute  outpatient  services.  Area  programs, 
DSS  departments,  and  mental  health  and  substance  abuse 
service  providers  must  continue  to  be  flexible  and  creative 
as  they  are  forced  to  adapt  the  service  system  to  new 
financing  approaches.  Hopefully  in  the  mission  to  improve 
both  efficiency  and  effectiveness  of  treatment,  CA  will 
continue  to  increase  access  and  delivery  of  a broad  spectrum 
of  community-based  services. 

Stakeholders'  Perspectives:  Client  Satisfaction 
and  Outcomes 

Elizabeth  M.Z.  Farmer,  Ph.D.  & Julia  S.  Gagliardi 

Children  and  families  who  use  the  public  mental-health 
system  are  the  most  important  stakeholders  in  CA.  They  are 
the  ones  for  whom  CA  was  created  and  the  ones  with  the 
most  to  gain  or  lose  depending  on  the  success  of  the 
program.  A set  of  measures  was  developed  and  pilot  tested 
to  explore  both  client  satisfaction  and  to  assess  child 
outcomes. 

Method 

A team  of  individuals&endash;composed  of  representatives 
from  the  state  Division  of  MH/DD/SAS,  Area  Programs, 
Family  advocacy  groups  (Families  Can  and  AMI),  and 
Duke  University&endash;developed  the  measures.  The 
group  focused  their  efforts  on  developing  measures  that  met 
several  core  requirements.  The  measures  must  assess 
satisfaction  and  outcomes  (process  measures  were  already 
under  discussion  elsewhere),  should  be  simple  to  both 
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administer  and  interpret,  had  to  tap  "real  world"  dimensions, 
and  should  be  useful  to  a variety  of  stakeholders  (e.g.,  the 
local  Area  Mental  Health  Programs,  the  State,  legislative 
committees).  The  committee  recognized  that  the  initial 
measures  would  be  the  first  step  in  an  ongoing  process  of 
development. 

The  child  outcome  measures  included  assessment  of 
problems  and  functioning  in  six  domains  (i.e.,  (a) 
school/work/vocational  training;  (b)  family/residential;  (c) 
peer  relations;  (d)  behavior;  (e)  substance  use;  and  (f) 
involvement  with  the  legal  system).  It  also  included  a 
checklist  of  treatment  foci  and  the  Child  and  Adolescent 
Functional  Assessment  Scale  (CAFAS;  Hodges,  Bickman, 

& Kurtz,  1991). 

The  satisfaction  measures  were  designed  to  gather 
information  at  key  points  in  a child's  treatment  process  (i.e., 
at  first  contact  with  the  center,  at  initial  treatment  planning, 
and  at  annual  update  of  the  treatment  plan  or  planned 
termination  of  treatment).  Each  of  these  forms  focused  on 
issues  that  were  of  particular  relevance  at  the  given  stage  of 
treatment.  In  addition,  a measure  was  designed  to  assess  the 
satisfaction  of  families  who  dropped  out  of  treatment. 

Pilot  data  on  the  satisfaction  measures  were  collected  in 
nine  Area  Programs  that  were  participating  in  CA.  Data 
collection  continued  for  one  month  and  included  all  children 
and  parents/guardians  who  met  the  criteria  for  the 
questionnaires  (e.g.,  all  clients  who  had  a first  contact, 
clients  who  had  annual  updates  of  treatment  plans). 
Questionnaires  were  given  to  both  a parent/guardian  who 
accompanied  the  child  and  to  the  child  (for  children  who 
were  at  least  ten  years  old). 

Questionnaires  were  completed  while  the  family  was  still  in 
the  clinic  and  were  returned  in  an  envelope  or  to  a "drop 
box"  to  insure  confidentiality.  Satisfaction  of  clients  who 
dropped  out  of  service  were  collected  via  telephone 
interviews  with  the  parent/guardian. 

Results 

A total  of  275  forms  were  included  in  the  analysis.  Of  these, 
1 15  were  "first  contact"  forms,  68  reported  on  satisfaction  at 
"initial  treatment  planning,"  53  reflected  views  at  annual 
review  or  planned  termination,  and  39  were  completed  by 
parents/guardians  of  children  who  had  recently  dropped  out 
of  treatment.  Available  data  suggests  a completion  rate  of 
approximately  79%,  though  this  rate  varied  considerably 
across  programs. 

The  following  results  highlight  areas  of  particular 
satisfaction,  as  well  as  areas  that  showed  room  for 
improvement.  At  first  contact,  99%  of  respondents 
expressed  adequate  satisfaction  with  their  experience  (i.e., 
75%  were  very  satisfied  and  24%  were  somewhat  satisfied). 


91 


Eighty-one  percent  reported  that  they  were  seen  at  the  Area 
Program  within  two  weeks  of  first  calling  to  make  an 
appointment,  and  91%  felt  that  this  response  time  was 
satisfactory.  Sixty-eight  percent  reported  that  they 
experienced  no  barriers  or  difficulties  in  obtaining  services. 
Of  those  who  did  report  difficulties,  the  most  commonly 
reported  problems  were  lack  of  transportation,  lack  of 
information,  and  concerns  about  costs.  Other  questionnaires 
continued  to  show  an  overall  high  level  of  satisfaction  with 
services.  Areas  of  particularly  high  satisfaction  included 
communication  (e.g.,  "staff  members  listened  to  what  you 
said,"  "staff  understood  the  needs  of  your  child  and 
family"),  and  efficacy  of  treatment  (e.g.,  "treatment  helped 
you  deal  more  effectively  with  problems").  Areas  that 
showed  a need  for  improvement  included  parental 
participation  in  treatment  planning  and  removing  barriers  to 
care  to  prevent  families  from  dropping  out  of  treatment. 

Pilot  testing  of  the  outcome  measure  included  28  clinicians 
or  case  managers  who  completed  the  assessment  on  41 
active  cases.  Results  indicated  that  the  forms  were 
acceptable  to  staff  members,  covered  domains  that  they 
considered  to  be  important,  and  could  be  completed  in  less 
than  ten  minutes. 

The  involvement  of  representatives  from  a variety  of 
perspectives  and  organizations  in  the  development  of  the 
satisfaction  and  outcome  measures  increased  interest  in  the 
measures  and  willingness  to  implement  them.  Pilot  testing 
showed  that  the  satisfaction  and  outcome  measures  were 
acceptable  to  clients  and  staff  members,  could  be  completed 
quickly  with  reasonable  return  rates,  and  gathered 
information  that  was  of  interest  and  use  to  the  intended 
stakeholders. 
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Stakeholders'  Perspectives:  Preliminary  Cost 
Findings 

David  Langmeyer,  Ph.D. 

This  portion  of  the  symposium  presents  a comparison  of 
service  costs  between  Area  Programs  that  participated  in 
CA  and  those  that  did  not.  Comparisons  within  each  group 
across  time  are  also  presented. 

Method 
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During  the  first  two  years  of  CA,  the  1 0 participating  Area 
Programs  were  paid  a capitated  rate  for  inpatient  services 
and  a fee-for-service  rate  for  non-inpatient  services,  while 
the  3 1 non-participating  Area  Programs  were  paid  a 
fee-for-service  rate  for  both  inpatient  and  non-inpatient 
services.  Hence,  all  costs  presented  here  were  calculated 
based  on  fee-for-service  rates.  For  this  report,  costs 
represent  the  Medicaid  rate  paid  for  services.  It  is  the  cost  to 
Medicaid  payers  (Federal,  State  and  County)  on  a 
fee-for-service  basis. 

Information  about  the  costs  under  CA  was  gathered  from 
the  North  Carolina  Medicaid  paid  claims  files  for  all  Area 
Programs  in  1992  and  the  non-participating  Area  Programs 
in  1994.  For  participating  Area  Programs,  1994  and  1995 
information  came  from  reimbursement  reports  of  the  North 
Carolina  Division  of  Mental  Health,  Developmental 
Disabilities,  and  Substance  Abuse  Services. 

It  is  known  that  there  is  under  reporting  on  the  paid  claims 
files.  Not  all  services,  particularly  non-inpatient  services, 
are  reported.  This  underestimates  the  cost  of  non-inpatient 
services  for  both  non-participating  and  participating 
programs  in  1992,  as  well  as  for  non-participating  programs 
in  1994.  The  result  is  that  cost  increases  for  participating 
programs  is  exaggerated. 

Results 

From  1992  (before  capitation)  to  1994  (first  year  of 
capitation),  total  value  of  services  for  participating  Area 
Programs  went  up  by  72%  (see  Table  IT  Non-participating 
Area  Programs  operating  under  fee-for-service  increased 
total  value  of  service  by  under  4%. 

Some  of  the  increase  was  due  to  an  increased  number  of 
eligible  children  under  18.  For  CA  participating  Area 
Programs,  there  was  a 26%  increase  in  eligible  months 
between  1992  and  1994.  For  non-participating  Area 
Programs,  this  increase  was  22%.  Combining  total  value  of 
service  and  number  of  eligibles,  the  cost  per  eligible  month 
increased  for  participating  Area  Programs  from  1 992  to 
1994  and  decreased  for  non-participating  Area  Programs  in 
the  same  period  of  time.  Participating  Area  Programs 
increased  cost  per  eligible  month  by  36%  and 
non-participating  Area  Programs  decreased  cost  per  eligible 
month  by  5%. 

In  general,  CA  was  very  successful  in  reducing  costs 
associated  with  inpatient  services.  From  1992  to  1994,  the 
cost  per  person  served  in  inpatient  settings  dropped  by  45% 
from  $14,976  per  person  to  $8,249.  Non-participating  Area 
Programs  started  at  about  the  same  level  as  participating 
Area  Programs  in  1992  and  increased  slightly  in  1994 
(+2%). 

Offsetting  the  decrease  in  inpatient  service  was  a dramatic 
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increase  in  non-inpatient  service  for  participating  Area 
Programs.  Value  per  person  served  in  non-inpatient  settings 
rose  from  $785  to  $2,552,  a 225%  increase.  There  was  also 
a sharp  increase  in  non-participating  Area  Programs'  value 
per  person  served  from  $603  to  $1,466  (+143%). 

The  decrease  in  inpatient  services  and  increase  in 
non-inpatient  services  about  balanced  out.  The  total  value 
per  person  served  for  participating  Area  Programs  in  1 992 
was  $3,203.  This  went  up  very  slightly  in  1994  ($3,280). 

The  non-participating  Area  Programs  were  comparable 
(-4.3%)  in  1992  (1994  was  not  available  at  time  of  the 
report). 

Aside  from  increasing  the  amount  of  service  provided, 
another  feature  of  CA  was  to  increase  availability  of 
services.  Evidence  supporting  this  accomplishment  is 
provided  by  the  fact  that  in  1994,  participating  Area 
Programs  served  7.5%  of  eligible  children  compared  to 
non-participating  Area  Programs  serving  4.8%  of  eligible 
children. 

The  reduction  in  inpatient  services  was  expected,  and  in 
fact,  was  the  basis  of  undertaking  CA  in  the  first  place,  but 
the  dramatic  increase  in  non-inpatient  services  needs  to  be 
explored.  One  aspect  of  the  increase  is  in  the  mix  of  services 
which  were  provided.  In  1 994,  over  60%  of  the 
non-inpatient  service  dollars  went  for  "High  Risk 
Intervention."  This  service  barely  existed  (as  a billable  • 
service)  in  the  1992  paid  claims  files.  A new  service 
accounted  for  the  increase  in  total  value  of  non-inpatient 
services.  Non-participating  Area  Programs  seem  to  be 
following  the  same  pattern  as  CA  programs  in  the  growth  of 
non-inpatient  services,  but  with  a lag  of  a few  months.  The 
month-to -month  paid  claims  for  non-inpatient  events  in 
non-participating  Area  Programs  grew  rapidly  from  July 
1994  until  March  1995  (1.5  million  to  3 million).  It  is  likely 
that  the  differences  in  non-inpatient  billing  between 
participating  and  non-participating  Area  Programs  will 
decrease  when  1 995  and  1 996  information  becomes 
available. 
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Managed  care  is  becoming  the  major  mechanism  for  the 
allocation  and  management  of  services  and  resources,  not 
only  for  private  health  care,  but  throughout  populations 
traditionally  served  by  public  service  systems.  This 
technology  has  moved  into  mental  health  services  and  is 
becoming  the  future  of  all  child  and  family  serving  systems, 
including  child  welfare  (Valentine,  Fisher,  Feild,  Webman, 
& Web,  1995).  This  has  led  many  communities  to 
experiment  with  the  key  principles  of  managed  care  before 
converting  to  managed  care  in  a formal  sense.  The  summary 
describes  a method  of  managing  resources  that  builds  on 
these  principles  in  order  to  allocate  a pool  of  blended, 
flexible  dollars  in  Ohio's  largest  metropolitan  county. 

History 

Franklin  County  has  a rich  heritage  of  intersystem  and 
public/private  collaboration.  Individualized  planning  and 
creative  funding  have  been  important  facets  of  the  service 
system  for  high  risk  children  and  youth  through  Kids  In 
Different  Systems  (KIDS),  the  local  Child  and  Family  First 
Intersystem  Council,  since  1992.  The  Council  is  comprised 
of  the  local  child  and  family  serving  systems  along  with 
provider  and  parent  representation  (see  Figure  1). 

Through  a pilot  project  called  the  10  Kids  Project,  KIDS 
experimented  with  bundled  contracting  through  a panel  of 
private,  non-profit  mental  health  service  providers.  In  this 
project,  ten  youths  in  out-of-county  placements  were 
brought  back  to  their  community  through  individualized 
planning  and  collaborative  implementation  (Cauble,  et  ah, 
1992).  This  project  successfully  demonstrated  the 
effectiveness  of  collaborative  planning  when  supported  by 
flexible  funding  for  serving  high  risk,  high  use  youth  with 
multiple  needs.  It  also  demonstrated  better  outcomes  and 
cost  efficiency  by  serving  them  in,  or  near  their  own  homes. 
A second  project  demonstrated  similar  results  with  a group 
of  multi-need  children  with  mental  retardation  served  by 
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one,  for-profit  agency.  In  this  project,  the  concept  of  limited 
risk  contracting  was  formalized. 

Objectives  of  Flexible  Funding 

Since  late  1991,  Franklin  County  has  been  committed  to  the 
flexible  allocation  of  blended  dollars  to  support 
individualized  plans  that  are  strength  based  and  family 
centered.  The  following  values  were  fundamental  in  the 
development  of  this  process: 

• Locus  of  decision  making  at  the  family  team  level 
(family  centered). 

• Optimum  flexibility. 

• Community  flexibility. 

• Accountability/tractability. 

• Outcome  oriented. 

• Sensitivity  to  family  satisfaction. 

Objectives  of  Managing  Care 

The  advent  of  managed  care  has  created  new  demands  on 
the  allocation  of  flexible  funds.  The  challenge  in  Franklin 
County  has  been  to  capitalize  on  the  benefits  of  managed 
care  for  high-risk,  intersystem  children,  youth,  and  families 
without  compromising  the  gains  made  in  the  area  of 
flexible,  nontraditional  individualized  service  delivery.  The 
following  principle  tenets  of  managed  care  were 
incorporated: 

• Limited  risk  contracting  (Lindstrom,  1994). 

• Increased  flexibility  for  the  provider  (provider  level 
risk  management  capabilities). 

• Increased  clinical  flexibility  and  responsibility  for  the 
provider. 

• Simplified  standardization  of  (a)  funding  mechanism 
for  individualized  plans,  (b)  tracking  of  individualized 
planning,  and  (c)  accounts  servicing  capabilities. 

• Improved  outcome  and  family  satisfaction  tracking, 
and  feedback 

• Increased  risk  management  capabilities  for  the 
funders 
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Project  Description 

Structure  of  Contracting 

The  issues  mentioned  above  create  the  value  base  on  which 
the  limited  risk  bundled  contracting  model  employed  in 
Franklin  County  is  built.  Contracting  is  done  with 
individual  providers  and  with  a local  Preferred  Provider 
Organization  (PPO).  The  contract  itself  coordinated  funding 
agreements  based  on  individual  plans  and  budgets 
developed  by  the  Child  and  Family  team  (CFT).  After  a 
plan  and  budget  review  process,  the  contract  is  written  for 
the  sum  of  the  approved  funding  agreements  with  the 
particular  agency  of  PPO  (Figure  2 presents  a flow  of  the 
contract  process). 

The  contract  allows  the  Lead  Agency  flexibility  in  using  the 
grand  total  across  individual  plans.  Hence  the  case  manager 
has  management  capacity  within  the  plan  across  line  items 
(life  domains),  and  the  agency  has  larger  risk  management 
capacity  across  plans  covered  in  the  contract.  In  turn,  the 
agency  is  responsible  to  provide  (or  contract  for  provision) 
and  coordinate  all  services  that  are  determined  to  be  needed 
by  the  CFT.  Since  all  planning  and  budgeting  is  a team 
centered  activity,  changes  in  services  and  expenditures  are 
discussed  in  monthly  team  meetings  and  reported  to  the 
funder  through  meeting  minutes,  which  are  also  published 
for  all  team  members. 

Upon  the  completion  of  the  contract,  which  is  renewed 
January  1 and  July  1 of  each  year,  the  Lead  Agency  submits 
a Reconciliation  which  reports  budget  expenditures,  actual 
receipt,  and  actual  expenditures  for  each  case  and  the 
contract  as  a whole.  Payment  is  made  in  two  quarterly 
installments  over  the  course  of  a six  month  contract.  The 
finalization  of  the  funding  agreements  and  the  signing  of  the 
contract  triggers  the  first  payment  in  advance.  This  front 
loading  is  another  strategy  to  support  the  providers  ability  to 
provide  needed  services  without  the  concerns  of  cash  flow, 
and  to  better  manage  their  risk.  The  second  payment  is 
triggered  by  the  receipt  of  all  required  submissions  for  the 
first  quarter. 

Limited  Risk  Structure 

The  contract  is  designed  to  provide  some  incentive  for  cost 
savings  without  creating  the  impetus  to  avoid  the  delivery  of 
needed  services.  Monitoring  through  required  submissions 
and  a conflict  resolution  procedure  supply  additional  points 
of  accountability.  The  contract  details  a profit/loss  window. 
If  actual  expenses  fall  under  the  budgeted  amount  by  up  to  a 
designated  percentage,  the  agency,  or  PPO  keeps  that 
amount.  All  monies  saved  beyond  the  window  are  returned 
to  the  funder.  Likewise,  if  actual  expenses  go  above  the 
budgeted  amount,  up  to  the  designated  percentage,  the  Lead 
Agency  is  responsible  for  continued  services  provision 
without  additional  dollars.  All  additional  expenses  above 
the  window  are  negotiated  for  reimbursement. 
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the  window  are  negotiated  for  reimbursement. 

Reporting  and  Monitoring 

Fiscal  information  is  provided  mainly  at  the  start  and  end  of 
the  contract  period.  Beyond  this  fiscal  information,  other 
reporting  requirements  are  purposefully  kept  at  a minimum 
in  order  to  improve  access  to  and  the  facilitation  of  services 
provided  on  the  basis  of  flexible  dollars.  One  required 
submission  is  team  meeting  minutes.  The  team  is  required 
to  meet  at  least  once  every  month  to  discuss  progress, 
needed  service,  and  budgetary  changes.  These  changes, 
along  with  other  pertinent  information,  are  recorded  in  team 
meeting  minutes  in  the  interest  of  smooth  communications 
and  improved  team  functioning.  These  minutes  are  sent  to 
KIDS  as  documentation  of  the  team's  ongoing  planning  and 
service  provision,  and  changes  in  services  and/or  budgeting. 
The  contract  also  requires  submission  of  documentation  of 
the  tracking  of  seven  behavioral  indicators  on  a daily  basis. 

Family  Satisfaction  and  Dispute  Resolution 

Another  facet  of  monitoring  contracted  services  is 
coordinated  directly  from  the  KIDS  office.  Family 
satisfaction  surveys  are  conducted  on  a quarterly  basis.  The 
survey,  developed  through  a series  of  parent  focus  groups, 
provides  a direct  link  between  funder  and  consumer.  The 
contract  indicates  that  the  Lead  Agency  will  be  notified  if 
family  satisfaction  levels  fall  below  a certain  point.  This 
notification  is  intended  as  the  completion  of  a necessary 
circle  of  communication  connecting  the  funder,  the 
provider,  and  the  consumer.  Such  communication  builds 
trust,  which  is  the  necessary  element  in  any  real  quality 
improvement  process.  Complaints  regarding  service 
provision  coming  from  participating  systems  are  handled  in 
similar  non-threatening  ways  through  procedures  internal  to 
KIDS,  but  developed  in  conjunction  with  the  PPO.  The 
major  thrust  is  on  communication  links  and  trust.  If  the 
complaints  persist,  a more  formal  resolution  conference  may 
be  held. 

Conclusion 

On  a small  scale,  Franklin  County  has  experimented  with 
managed  care  principles  based  on  individualized  planning 
that  is  strength  focused,  needs  driven,  and  family-centered. 
These  experiments  have  led  to  a method  of  contracting  that 
appears  to  maintain  the  integrity  of  individualized  services, 
flexible  funding,  and  financial  accountability,  while  creating 
a foundation  for  managing  services  on  a much  larger  scale. 
On  this  foundation,  we  can  move  in  a variety  of  directions, 
such  as  a leveling  or  case  rate  system,  as  we  develop  the 
best  structure  of  managed  care  for  our  community. 

Financially,  the  purpose  is  to  create  a situation  of  shared 
risk.  It  is  important  that  the  providers  actually  assume  risk, 
as  opposed  to  passing  the  cost  of  the  risk  on  by  inflating 
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service  and/or  overhead  costs.  It  appears  that  we  have  been 
successful  in  this  area  thus  far.  The  overhead  costs  have 
been  predetermined  through  a time  study  process  and  it  is 
evident  that  providers  have  not  offset  the  assumption  of  real 
risk  by  increasing  service  costs.  Figure  3 outlines  average 
daily  costs  before  and  after  the  implementation  of  bundled 
contracting.  The  increase  observed  at  the  time  of  bundles 
contracting  can  be  attributed  to  a change  in  Medicaid  billing 
rules  in  the  State  of  Ohio.  In  the  following  period,  the 
average  daily  costs  return  to  previous  levels.  Patterns  of 
inflation  in  services  costs  will  be  monitored  in  order  to 
determine  continuous  success. 

However,  the  success  of  this  process  of  managing  care 
through  limited  risk  bundled  contracting  can  not  be  judged 
solely  on  management  and  financial  criteria.  Innovations  in 
managed  care  cannot  be  made  at  the  expense  of  front  line 
flexibility  needed  to  serve  children  and  their  families  with 
high  level,  multiple  needs.  The  contracting  mechanism 
described  here  has  been  in  place  for  six  months.  Although  it 
is  too  early  to  be  conclusive,  the  initial  impressions  are 
favorable.  With  a multiple  of  changes  occurring  around  (a) 
federal,  state,  and  local  funding  strategies,  (b)  methods 
service  provision,  and  (c)  outcomes  collection,  it  is 
impossible  to  single  out  particular  effects  of  these  contracts 
with  extreme  clarity.  However,  we  have  anticipated  some 
impact  from  the  implementation  of  this  contracting.  For 
example,  we  can  look  at  behavioral  indicators  tracked 
during  the  service  period.  The  indicators  traced  are  alcohol 
and  drug  use,  suicide  attempts,  AWOLs,  assaultive 
behavior,  self-injurious  behavior,  missed  school,  and  arrests. 
During  this  contract  period  we  have  seen  a slight  increase  in 
the  number  of  days  containing  key  behavioral  indicators 
(see  Figure  4).  It  is  probable  that  the  stress  placed  on 
outcomes  in  the  contract  will  serve  to  increase  the  accracy 
of  target  behavior  measurement. 

We  also  anticipate  that  family  satisfaction  with  planning  and 
services  will  not  be  adversely  effected.  Further,  it  is  hoped 
that  this  way  of  managing  services  will  eventually  serve  to 
increase  family  satisfaction,  and,  conversely,  that  the  need 
for  outside  conflict/dispute  resolution  will  decrease. 

Only  through  successful  management  of  resources  can  we 
hope  to  make  individualized  service  approach  the  norm,  as 
opposed  to  the  exception.  We  believe  that  this  model  of 
contracting  provides  a solid  starting  point. 
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Introduction 


Over  the  past  three  to  five  years,  there  has  been  a rapid 
transition  to  managed  care  models  of  health  care  financing 
and  organization  in  the  United  States.  This  transition  is  now 
reaching  the  area  of  public  mental  health  services.  As  states 
and  governmental  entities  seek  to  control  the  increasing 
costs  of  Medicaid  programs,  many  are  moving  towards 
implementation  of  managed  care  principles  in  the  funding 
and  delivery  of  mental  health  services.  States  are  pursuing 
different  models  of  managed  mental  health  services  under 
both  the  1150  and  1915  Medicaid  waivers,  some  contracting 
with  established  managed  care  providers  and  others 
attempting  to  convert  components  of  their  public  systems 
into  managed  networks. 

This  transition,  however,  is  not  based  on  any  solid  data 
concerning  the  funding  needed  to  serve  populations 
traditionally  served  by  the  public  mental  health  system,  nor 
has  it  been  based  on  any  of  the  solid  conceptual  models  and 
principles  which  have  been  tested  with  these  populations. 
The  managed  care  models  which  have  been  typically 
implemented  by  managed  care  contractors  have  not  taken 
into  account  the  special  needs  of  Medicaid-covered 
populations.  These  special  needs  include  children  with 
severe  emotional  disturbances  and  other  disabilities, 
economically  distressed  families,  and  members  of 
under-served  minority  groups,  as  opposed  to  the  majority 
middle  class  populations  around  which  managed  care 
services  were  initially  developed. 

This  oversight  has  already  led  to  adverse  consequences  in 
some  state  managed  Medicaid  contracts  where  the  needs  for 
services  in  these  populations  (and  the  consequent  capitation) 
was  grossly  underestimated.  On  the  other  hand,  many  state 
mental  health  agencies  are  attempting  to  convert  into 
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managed  care  organizations  and  funding  of  services,  but 
often  without  the  infrastructure  support  for  a successful 
conversion. 

These  summaries  will  address  conceptual  and  infrastructure 
issues  which  both  traditional  managed  care  providers  and 
converting  state  mental  health  systems  must  face  to  make  an 
effective  transition  towards  a managed  care  organization  of 
services. 

Clinical  Experiences  in  Managed  Care 
Implementation  for  Children  with  Serious 
Emotional  Disturbances 

Theodore  Fallon,  M.D.,  M.P.H. 

The  Robert  Wood  Johnson  Foundation  funded  eight 
demonstration  sites  around  the  country  with  the  goal  of 
constructing  arrays  of  service  that  might  address  the  range 
of  needs  for  the  most  disturbed  children  and  youth  and  their 
families.  The  actual  structures  of  these  programs  varied 
considerably  and  were  each  suited  for  the  political,  social, 
and  clinical  climate  in  which  they  were  implemented.  This 
project,  the  Mental  Health  Service  Program  for  Youth, 
yielded  eight  examples  of  what  is  possible  given  adequate 
resources. 

Clinical  aspects  of  these  programs  were  examined  by 
observing  the  clinical  work  through  case  conferences  by 
visiting  clinicians.  The  success  of  these  programs  in 
working  with  these  disturbed  youth  and  their  families 
seemed  to  be  dependent  on  a number  of  attributes: 

1 . The  services  provided  impact  on  all  aspects  of  the 
youth  and  his  family's  life.  At  a clinical  level,  this 
translates  into  multiple  system  involvement  including 
juvenile  justice,  child  welfare,  education,  and  mental 
health.  On  a systems  level,  this  translates  into  pooled 
funding  of  these  agencies,  in  essence,  pooling  an 
array  of  resources  and  flexibly  determining  their  most 
efficacious  use.  An  example  of  this  was  seen  at  one 
site  when  a physically  large  adolescent  repeatedly 
threatened  the  residential  staff.  During  these  times, 
the  juvenile  justice  system  allowed  use  of  the 
detention  center  for  time  out  periods.  The  youth  was 
able  to  settle  down  enough  to  use  residential  support 
and  educational  services,  and  eventually  the  youth 
became  a good  student. 

2.  This  example  also  illustrates  the  utility  of  coordinated 
services. 

3.  Services  also  need  to  be  attractive.  Many  examples 
were  seen  in  which  youths  and  families  had 
repeatedly  rejected  services,  but  when  the  services 
offered  something  desirable,  families  used  them  and 
in  that  setting  were  able  to  develop  alliances  with 
workers. 

4.  The  services  that  were  the  most  successful  involved 
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4.  The  services  that  were  the  most  successful  involved 
considerable  creativity  in  the  service  of  being 
responsive  to  the  youth  and  family.  This  means  that 
the  clinicians  had  control  of  the  services  and  access  to 
experts  if  they  themselves  were  not  expert.  This 
expertise  involved  two  components:  (a)  knowledge 
and  practical  skills  in  engaging  and  holding  on  to  the 
families  and  children  (characterized  by  focusing  on 
the  child  and  families'  strengths,  building  rapport  and 
self  esteem);  and  (b)  knowledge  and  practical  skills  in 
assessing  and  working  with  deficits  including 
psychopathology  (necessary  in  order  to  know  how  to 
direct  the  rehabilitative  services  that  were  needed  to 
move  the  child  and  family  toward  normal 
development). 

5.  The  services  needed  to  have  sufficient  resources  to  be 
able  to  sustain  them  for  as  long  as  was  necessary. 

This  required  a commitment  on  the  part  of  the 
organizations  involved  (i.e.,  an  ability  to  withstand 
the  political  changes  as  well  as  the  personnel 
changes).  This  attribute  seemed  to  be  the  most 
difficult  to  attain,  especially  in  the  environment  of 
managed  competition  where  competition  implies 
competition  of  services  where  one  naturally  displaces 
another. 

These  five  attributes  were  present  to  some  degree  in  all  of 
the  systems  of  care  that  were  able  to  make  gains  in  working 
with  children  and  youth  with  serious  emotional 
disturbances. 

Best  Principles  for  Managed  Care  Request  for 
Proposals 

Andres  J Pumariega,  M.D. 

Children  and  adolescents  covered  by  Medicaid  often  have 
multiple  developmental  needs  and  complex  problems. 
Effective  intervention  thus  needs  to  include  a full  array  of 
services  in  a community-based  system  of  care,  based  on 
Child  and  Adolescent  Service  System  Program  (CASSP) 
principles.  The  American  Academy  of  Child  and  Adolescent 
Psychiatry's  Task  Force  on  Community  Systems  of  Care,  in 
collaboration  with  other  mental  health  professional  and 
service  associations  and  organizations,  has  developed  a 
document  titled  Best  Principles  for  Managed  Medicaid 
Request  for  Proposals. 

The  Best  Principles  for  Managed  Medicaid  Request  for 
Proposals  is  designed  to  assist  state  decision  makers  in 
selecting  managed  care  Medicaid  vendors  or  developing 
public  managed  systems  that  can  most  effectively  serve  the 
population  of  Medicaid  covered  child  population, 
particularly  multi-problem  children  with  severe  emotional 
disturbances.  It  defines  principles  that  should  be  inherent  in 
high  quality  programs.  It  addresses  such  issues  as 
governance  of  service  systems,  design  of  benefits,  access  to 
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services,  development  of  treatment  plans,  triage  and 
assessment,  treatment  services,  case  management,  quality 
assurance,  information  management,  and  provider  support 
by  states. 

Providing  a full  continuum  of  services  allows  the  care 
providers  to  customize  the  care  plan  to  most  effectively  help 
the  patient  and  family.  The  focus  of  such  a model  is  on 
managing  services  rather  than  managing  benefits  a priori, 
which  is  discriminatory  and  restrictive  with  under-served 
populations.  This  model  also  ensures  coordinated  care  and 
the  establishment  of  communication  and  collaboration 
across  disciplines  and  agencies  for  effective  coordination  of 
services.  Care  plans  must  be  patient  and  family  centered, 
with  their  full  involvement  in  the  assessment  and  treatment 
processes.  These  programs  also  need  to  develop  programs 
and  monitor  outcomes  in  the  following  areas:  (a)  access,  (b) 
functional  and  clinical  outcomes,  (c)  prevention,  (d) 
wellness,  (e)  community  acceptance/responsiveness,  and  (f) 
patient/family  satisfaction. 

This  document  has  been  forwarded  to  all  50  Medicaid 
directors,  and  has  received  overwhelming  positive 
responses,  including  responses  from  over  40  states 
requesting  more  information  and  consultation.  Information 
regarding  the  principles  covered  in  this  document,  as  well  as 
the  approaches  being  considered  to  evaluate  its  adoption  and 
implementation  can  be  obtained  from  the  author. 
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Introduction 


This  collection  of  summaries  presents  the  latest  analyses  of 
data  addressing  the  role  of  system  changes  in 
decision-making  and  service  utilization  in  child  and 
adolescent  (C/A)  mental  health  emergencies.  In 
Massachusetts,  private  management  of  mental  health  and 
substance  abuse  benefits  in  the  public  sector  began  in  1992. 
Since  then,  increased  management  of  these  benefits  has 
occurred  in  the  private  sector  as  well.  The  findings  reported 
in  this  symposium  are  part  of  on-going  efforts  to  evaluate 
private  sector  strategies  to  manage  public  sector  mental 
health  benefits. 

In  previous  analyses  (Young,  Simon,  Nicholson,  & 
Bateman,  1996),  the  Child  and  Adolescent  Service  System 
Program  (CASSP)  concepts  of  least  restrictive  environment, 
community-based  services,  and  individual  need  provided  a 
framework  for  exploring  the  contributions  of  demographic, 
clinical,  and  fiscal  variables  to  dispositions  in  emergency 
mental  health  screening  of  children  and  adolescents,  before 
and  after  the  private  management  of  Medicaid  benefits.  As 
expected,  the  volume  of  screening  episodes  increased  after 
managed  care  was  implemented.  There  was,  however,  a 
significant  decline  in  the  proportion  of  dispositions  to 
inpatient  hospital  settings,  suggesting  that  children's  needs 
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were  being  met  in  less  restrictive  settings  after  the 
implementation  of  the  managed  care  program. 

During  the  same  time  period,  there  was  a significant 
increase  in  the  number  of  dispositions  to  community-based 
crisis  stabilization  programs,  indicating  that  many  children 
who  would  have  been  hospitalized  were  being  diverted  to 
community-based  programs  after  the  implementation  of  the 
Medicaid  managed  care  program.  Those  emergency 
episodes  in  which  children  and  adolescents  were  in  greatest 
clinical  need,  however,  continued  to  result  in 
hospitalizations  after  the  implementation  of  managed  care, 
suggesting  that  children's  individual  needs  were  factored 
into  decision-making. 

In  this  symposium,  we  present  three  analyses  of  data  from 
C/A  recidivists  at  a mental  health  emergency  screening  site 
in  pre-  and  post-managed  care  time  periods,  defined  by  the 
implementation  date  for  Medicaid  managed  care.  These 
analyses  were  conducted  to  address  the  fears  of  many 
clinicians  that,  in  a managed  care  environment,  clients  are 
provided  brief,  "band-aid"  services  that  do  not  fully  address 
underlying  problems.  The  assumption  made  by  many 
clinicians  is  that  utilization  reviewers  are  reluctant  to 
approve  more  costly  services  that  clients  may  really  need,  or 
that  they  will  not  approve  them  until  less  costly  services 
have  been  tried.  The  common  concern  is  that  clients  will 
have  to  keep  coming  back  for  treatment  until  their  needs  are 
adequately  met. 

The  following  questions  were  addressed  regarding  the 
impact  of  managed  care  on  recidivists  at  an  emergency 
screening  site: 

• Was  there  a change  in  the  number  of  recidivists  and, 
specifically,  in  the  number  of  Medicaid  recidivists 
following  the  implementation  of  managed  care? 

• Which  clinical,  client,  and  fiscal  variables  predicted 
becoming  an  Emergency  Mental  Health  Services 
(EMHS)  recidivist  in  the  pre-  and  post-managed  care 
periods? 

• What  were  the  disposition  patterns  for  recidivists  in 
the  pre-  and  post-managed  care  periods? 
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Method 


Data  for  all  three  analyses  were  drawn  from  the  daily  log 
sheets  of  the  EMHS  program  at  the  University  of 
Massachusetts  Medical  Center  in  Worcester,  MA.  These 
logs  contain:  (a)  demographic  and  clinical  characteristics  for 
clients  receiving  services,  (b)  type  of  insurance,  (c)  referral 
source,  and  (d)  disposition. 

We  looked  at  clinical  and  fiscal  variables  and  dispositions 
for  children  and  adolescents  over  a 

2-1/2  year  period  from  October,  1991  to  March,  1994.  We 
divided  the  2-1/2  year  period  into  3 parts:  (a)  pre-managed 
care  (i.e.,  one  year  prior  to  the  implementation  of  managed 
Medicaid  mental  health  benefits  on  October  1 , 1 992);  (b)  a 
six  month  transition  period;  and  (c)  post-managed  care  (i.e., 
the  year  following  the  start-up  of  the  managed  Medicaid 
program).  We  did  not  include  the  transition  period  in  our 
analyses  to  avoid  drawing  conclusions  based  on  the  normal 
fluctuations  that  are  involved  in  any  major  systems  shift. 

Although  Medicaid  recipients  account  for  the  majority  of 
EMHS  episodes  across  the  2-1/2  year  period,  EMHS  also 
became  the  screening  site  for  a number  of  private  managed 
care  organizations  during  this  time.  Therefore,  the  research 
period  really  represents  a time  of  overall  systems  change. 

For  this  reason,  both  Medicaid  and  non-Medicaid  recidivists 
were  included  in  our  analyses,  unless  otherwise  noted. 

Impact  of  Managed  Care  on  the  Clinical 
Profiles  of  Recidivists 

Lorna  Simon,  M.A.  and  Stephen  Dine- Young,  M.A. 

In  this  summary,  demographic,  clinical,  and  insurance 
profiles  of  C/A  recidivists  in  the  pre-  and  post-managed  care 
periods  were  compared  using  chi-square  tests.  Again,  clients 
screened  in  the  transition  period  were  excluded. 

There  were  a total  of  482  children  and  adolescents  seen  in 
the  pre-managed  care  (10/1/91  to  9/31/92)  and 
post-managed  care  (4/1/93  to  3/30/94)  periods.  Of  these 
clients,  101  were  recidivists.  The  clinical  and  payor 
characteristics  of  the  recidivists  are  presented  in  Table  1 . 

The  gender  split  was  relatively  even.  A majority  of 
recidivists  were  adolescents,  and  most  either  lived  with 
family  or  at  a residential  treatment  facility.  The  difficulties 
that  recidivists  were  most  likely  to  present  with  were 
harmful  to  self  or  other  and  problem  behaviors  (i.e.,  "acting 
out");  these  difficulties  corresponded  to  frequent  diagnoses 
of  disruptive  D/0,  Post  Traumatic  Stress  Disorder 
(PTSD)/ Anxiety  D/0,  and  Adjustment  D/0.  Over 
two-thirds  of  the  recidivists  had  Medicaid  insurance,  while 
most  of  the  remaining  clients  had  some  form  of  private 
insurance  (either  indemnity  or  HMO). 
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Of  the  101  C/A  recidivists,  37  (17%  of  the  total  C/A  clients) 
were  screened  in  the  pre-managed  care  period,  and  64  (24% 
of  the  total  C/A  clients)  were  screened  in  the  post-managed 
care  period&endash;an  increase  that  was  statistically 
significant  (c2  = 4.01,  p < .05). 

Although  none  of  the  clinical  or  demographic  variables 
were  significant  across  the  two  time  periods,  there  was  a 
statistically  significant  increase  in  the  proportion  of 
recidivists  who  were  Medicaid  beneficiaries  in  the 
post-managed  care  period  (75%  versus  54%  in  the 
pre-managed  care  period;  c2  = 4.65,  p < .05). 

There  was  an  increase  in  the  number  of  emergency  room 
recidivists  after  the  implementation  of  managed  care.  This 
difference  can  be  attributed  to  the  management  of  benefits 
through  support  by  the  additional  finding  that  there  were  no 
differences  in  clinical  characteristics  in  the  pre-  and 
post-managed  care  periods,  suggesting  that  there  was  little 
change  in  the  client  population  across  the  two  time  periods. 
The  finding  that  a larger  proportion  of  the  recidivists  were 
Medicaid  clients  in  the  post-managed  care  period  can 
probably  be  attributed  to  new  regulations  requiring  that  all 
Medicaid  clients  be  screened  at  EMHS  before  receiving 
other  services. 

Factors  Contributing  to  a Child  or  Adolescent 
Becoming  a Recidivist  at  an  Emergency  Mental 
Health  Screening  Site  Pre-  and  Post-Managed 
Care 

Stephen  Dine-Young,  M.A.  and  Loma  Simon,  M.A. 

In  the  second  summary,  demographic,  clinical,  and 
insurance  variables  were  tested  as  predictors  of  C/A  clients 
becoming  recidivists  before  and  after  the  implementation  of 
managed  care. 

Method 

Logistic  regressions  for  the  pre-  and  post-managed  care  time 
periods  were  conducted  to  predict  the  odds  of  becoming  a 
recidivist,  as  determined  by  more  than  one  visit  to  EMHS 
during  the  time  period.  For  each  of  the  periods,  the 
following  predictors  were  considered:  age,  gender, 
diagnosis,  and  insurance.  Adjusted  odds  ratios  were 
calculated  for  each  predictor. 

Results 

The  results  of  the  logistic  regression  for  the  pre-managed 
care  period  are  presented  in  Table  2.  Only  a diagnosis  of 
adjustment  disorder  significantly  predicted  whether  a C/A 
client  became  a recidivist  in  the  pre-managed  care  period, 
and  the  relationship  was  negative  (B  = -1 .69;  p < .01).  A 
child  diagnosed  with  this  a disorder  was  less  than  one-fifth 
as  likely  to  be  an  EMHS  recidivist  than  a child  diagnosed 
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as  likely  to  be  an  EMHS  recidivist  than  a child  diagnosed 
with  disruptive  D/0. 

The  results  of  the  post-managed  care  logistical  regression 
are  presented  in  Table  3.  Again,  only  the  coefficient  for 
adjustment  disorder  was  significant  and  the  relationship  was 
negative  (li  = -1.04;  p < .05).  In  addition,  being  a Medicaid 
client  also  significantly  predicted  whether  a C/A  client 
would  become  a recidivist  (li  = 1 .42;  p <.00 1).  Medicaid 
clients  were  more  than  4 times  as  likely  to  return  to  EMHS 
than  non-Medicaid  clients. 

It  also  should  be  noted  that  in  the  pre-managed  care  period 
there  was  a trend  toward  significance  for  mood  disorders  (li 
= 1.87;  p < .1)  and  PTSD/ Anxiety  disorders  (li  = .98;  p < 

.1).  These  trends  were  not  evident  in  the  post-managed  care 
period. 

Conclusions 

The  finding  that  adjustment  disorder  negatively  predicted  a 
C/A  EMHS  client  becoming  a recidivist  in  both  the  pre-  and 
the  post-managed  care  periods  is  understandable  in  that  it  is 
the  least  severe  of  any  of  the  diagnostic  categories;  clients 
assigned  adjustment  disorder  diagnoses  are  probably  less 
likely  to  need  repeated  care.  Given  the  post-managed  care 
period  requirement  that  Medicaid  recipients  be  screened  at 
EMHS,  the  finding  that  having  Medicaid  insurance 
significantly  predicted  a C/A  client  becoming  a recidivist 
also  is  not  surprising. 

Clinicians  may  be  concerned  about  the  trend  toward 
significance  with  the  diagnoses  of  mood  disorder  and 
PTSD/anxiety  disorder  in  the  pre-managed  care  period  that 
did  not  exist  in  the  post  period.  While  these  findings  are  not 
strong  enough  to  draw  clear  interpretations,  there  is  some 
indication  that  clinical  factors  are  being  given  less 
consideration  in  the  post-managed  care  period.  Further 
investigation  is  necessary. 

The  Question  of  Patterns  of  Dispositions 

Joanne  Nicholson,  Ph.D.  and  Joseph  R.  Mara,  B.A. 

In  this  summary,  levels  of  restrictiveness  of  dispositions  for 
C/A  mental  health  emergency  screening  recidivists  in  the 
pre-  and  post-managed  care  period  were  considered.  In 
previous  analyses  (Young  et  al.,  1996),  it  was  found  that 
although  emergency  screening  volume  increased 
significantly  in  the  post-managed  care  period,  admissions  to 
inpatient  settings  significantly  decreased.  In  addition,  this 
decrease  seemed  to  be  in  direct  proportion  to  the  volume  of 
episodes  resulting  in  referrals  to  newly  developed  crisis 
stabilization  programs  (e.g.,  community-based  care).  It  was 
assumed  that  an  increase  in  referrals  to  community-based 
services  was  a good  thing. 
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There  are  questions,  however,  that  may  nag  certain 
stakeholders  (i.e.,  both  providers  and  consumers).  Does  this 
increase  in  referrals  to  community -based  services  reflect  a 
lack  of  access  to  the  appropriate  level  of  care  for  those 
children  and  adolescents  in  greatest  clinical  need?  Also,  do 
children  keep  coming  back  until  their  needs  are  ultimately 
addressed  via  the  inpatient  level  of  care  they  really  needed 
in  the  first  place?  If  this  is  the  case,  costs,  both  dollar  and 
clinical,  may  only  be  delayed  and  higher  as  an  end  result. 

Method 

We  developed  a very  simple  coding  scheme  for  describing 
the  pattern  of  dispositions  for  C/A  mental  health  screening 
recidivists.  We  examined  the  dispositions  for  recidivists 
over  time,  and  if  the  level  of  restrictiveness  of  the 
dispositions  remained  the  same,  a code  of  "no  change"  or 
"="  was  assigned.  If  the  level  of  restrictiveness  of  the 
dispositions  decreased  over  time,  this  was  coded  as 
"decrease"  or  If  the  level  of  restrictiveness  of  the 

dispositions  increased  over  time,  this  was  coded  as 
"increase"  or "+." 

There  is  a weakness  in  this  strategy;  change  over  time  was 
condensed  into  one  code.  In  doing  this,  important 
information  was  lost  that  may  have  contributed  to  changes 
in  dispositions  over  multiple  emergency  screening  episodes 
(e.g.,  changes  in  age  of  the  child  or  adolescent  and  related 
changes  in  available  service  options  and  the  developing 
course  of  a disturbance  or  disability). 

Chi-square  tests  were  used  to  determine  if  there  were 
significant  differences  between  pre-  and  post-managed  care 
periods  in  patterns  of  dispositions. 

Results 

None  of  our  independent  variables  (i.e.,  clinical  and  payor 
characteristics)  were  related  to  change  in  level  of 
restrictiveness  of  dispositions  in  the  pre-  or  post-managed 
care  periods,  either  for  the  total  group  of  recidivists  or  the 
Medicaid  subscribers  who  were  recidivists.  It  would  seem 
that  the  implementation  of  managed  care  had  no  impact  on 
the  pattern  of  restrictiveness  of  dispositions  for  this  sample, 
as  far  as  our  simple  coding  scheme  could  detect. 

Discussion 

One  way  to  think  about  this  finding  is  that  children  are 
continuing  to  receive  the  care  they  need.  There  may  be 
something  about  their  ages  or  stages  of  illness  that  is  more 
powerful  than  the  payor  variable  in  determining  disposition. 

There  was  no  increase  in  numbers  of  children  and 
adolescents  in  the  increasing  level  of  restrictiveness  group, 
which  would  have  suggested  that  children  were  being  kept 
from  the  level  of  care  they  required  until  the  gatekeepers. 


113 


despite  their  efforts  to  contain  costs,  could  no  longer  deny 
them  access.  Rather,  we  still  see  the  same  proportions  of 
children  for  whom  alternative  plans  are  tried  before  they  are 
hospitalized.  And,  even  if  they  are  eventually  hospitalized, 
there  may  be  some  clinical  benefit  to  trying  alternatives 
first. 

What  is  clear  is  the  need  for  coordinated  evaluation  efforts 
among  public  and  private  sector  agencies  and  providers  if 
we  are  to  truly  understand  the  impact  of  managed  care  on 
such  important  issues  as  decision-making,  service  access 
and  quality  of  care. 
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A Community-Based  Continuum  of  Services 
Compatible  within  the  Managed  Care  Environment 

1 

Authors 

Community-based  Treatment  Hubs 


The  Spurwink  School  offers  a series  of  treatment  "Hubs" 
that  are  positioned  in  a tri-state  area.  This  summary  outlines 
the  "Hub"  model  and  its  key  features. 


Administrative  Structure 


The  continuum  of  services  within  the  Spurwink  system 
involves  commimity  based  programming  in  a wide 
geographic  area  in  the  states  of  Maine,  New  Hampshire,  and 
Rhode  Island.  This  involves  a central  administrative 
structure  as  well  as  decentralized  program  supervision 
structures.  The  central  administration  is  involved  in  such 
[;  activities  as  financial  administration  and  accountability, 

» program  coordination,. training,  governance,  computerized 

I systems,  human  resources  coordination,  clinical  and 

I educational  oversight,  quality  assurance,  licensing, 

ti  accreditation,  government  relations  and  fundraising.  The 

» decentralized  supervision  is  based  on  the  use  of  program 

I directors  who  provide  oversight  to  clusters  of  programs 

j1  which  are  based  upon  the  categories  of  services  and  clients, 

W the  geographic  location  or  the  setting  in  which  services  are 

|j  provided. 


Within  the  children  and  adolescent  segment  of  the  program, 
the  geographically  specific  day  treatment  centers  serve  as 
"Hubs"  for  the  day  and  residential  programming;  each  has  a 
program  director.  Program  directors  also  serve  an  integral 
function  at  the  adult  developmental  disabilities  day  activity 
and  residential  programs,  the  Spurwink  Clinic  and  the  two 
residential  treatment  centers,  including  the  specialized 
adolescent  and  the  long-term  staff  secure  programs.  These 
program  directors  are  the  vehicle  for  communication, 
quality  assurance,  training,  financial  accountability, 
treatment,  education,  and  various  other  activities.  This  entire 
model  involving  a central  administrative  structure  and  a 
decentralized  program  supervision  structure  managed  by  a 
program  director  assures  integration  of  services  and 
accountability. 
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The  Generalist  Model 

The  core  of  the  treatment  activities  is  via  the  professional 
identified  as  "Generalists."  The  generalist  became  the  core 
of  the  treatment  paradigm  employed  at  Spurwink;  and 
thirty-five  years  later,  across  three  states  and  more  than  60 
sites,  it  remains  the  philosophical  approach;  and 
importantly,  the  key  to  its  managed  care  activities. 

This  generalist  model  was  developed  from  the  recognition 
that  child  care  workers  and  teachers  are  the  core 
professionals  most  exposed  to  the  challenging  behavior  of 
kids  with  emotioonal  or  behavioral  disturbances. 

In  order  for  these  front-line  workers  to  be  effective, 
immediate,  and  responsive,  support  services  and  supervision 
are  required.  This  is  the  role  of  the  generalist  - the  one  who 
integrates  and  coordinates  all  that  goes  on  in  the  life  of  a 
particular  child.  The  generalist  is  responsible  for  working 
with  the  child's  family,  working  with  the  clinical  staff, 
working  with  child  care  workers,  and  working  with 
teachers.  The  task  is  to  communicate,  integrate,  translate, 
support,  and  supervise. 

The  generalist  model  recognizes  the  fact  that  fragmented 
children  and  families  cannot  effectively  be  treated  with 
fractionalized  services. 

The  fascinating  reality  of  this  model  has  been  its 
adaptability.  Established  as  a key  component  of  residential 
treatment,  the  generalist  has  been  used  effectively  in  day 
treatment,  adult  intermediate  care  facilities  and  community 
based  residences  designed  to  bring  youngsters  out  of 
intensive  in-patient  facilities. 

With  this  as  its  philosophical  core,  it  is  easy  to  see  why 
Spurwink  began  to  expand  its  mission  in  a decentralized 
fashion.  Each  community  that  was  developed  was 
designated  as  a "Hub."  For  every  hub  there  is  a series  of 
residential  alternatives,  a day  treatment  site,  a cooperative 
relationship  with  a public  school  and  a network  of 
consultants  and  providers,  ranging  from  psychiatry  to 
nutritionists  to  vocational  training  specialists.  Each 
community  "Hub"  and  all  of  its  components  are  linked 
together  and  held  accountable  by  the  Generalist,  the 
treatment  "Hub." 

Given  the  least  restrictive  to  most  restrictive,  definition  of 
the  residential  continuum,  Spurwink  was  able  to  respond 
quickly  to  being  able  to  accept  difficult  to  place  and  difficult 
to  manage  youngsters.  This  also  permitted  intra-agency 
movement  of  youngsters,  if  necessary.  Because  of  this 
Spurwink  developed  the  reputation  of  being  flexible  and 
adaptive.  Of  course,  this  is  a core  requirement  within  the 
managed  care  paradigm. 
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Accountability  & Information  Systems 
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A growing  sense  of  urgency  regarding  the  need  to  reform 
present  patterns  of  delivering  human  services  to  children 
and  families  underscores  the  need  for  systems  to  have 
mechanisms  that  ensure  accountability.  One  strong  element 
of  this  reform  is  that  human  services  need  to  become 
accountable  for  achieving  measurable  outcomes  rather  than 
continuing  to  focus  on  technical  compliance  with  rules  or 
on  simple  demonstration  of  service  need.  Doug  Nelson, 
Executive  Director  of  the  Annie  E.  Casey  Foundation  notes, 
"It  has  become  a well-worn  observation  that  success  in 
human  services  is  too  often  measured  by  persons  served  or 
services  provided  and  too  rarely  by  results  achieved. 
Difficult  though  it  may  be,  the  reform  required  is  clear. 
Helping  agencies,  service  programs  and  schools  need  to  be 
held  genuinely  accountable  for  progress  on  specific, 
publicly  articulated  and  accurately  tracked  outcomes  for  the 
children  and  families  they  serve."  (Nelson,  1993).  The 
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summaries  that  follow  describe  an  evolutionary  process  for 
incorporating  use  of  outcome  information  into  planning  and 
program  development. 

The  opening  summary  of  this  symposium  presents  the 
Ecology  of  Outcomes  framework,  an  overall  framework 
designed  to  guide  the  development  of  outcome  information 
systems.  This  framework  provides  a foundation  for  utilizing 
outcome  information  in  ways  that  provide  opportunities  for 
learning  and  self-correction;  it  emphasizes  using  outcome 
information  to  inform  decisions  that  shape  service  planning 
and  delivery.  As  discussed  below,  components  of  the 
Ecology  of  Outcomes  framework  include  the  Principles  for 
Outcome  Accountability,  Prerequisites  and  Building  Blocks 
for  building  outcome  accountability,  guidelines  for 
Implementing  an  Outcome  Information  System,  and  an 
approach  for  Utilizing  the  Results.  This  framework  was 
developed  through  the  System  Accountability  Study,  an 
initiative  of  the  federally  funded  Research  and  Training 
Center  for  Children's  Mental  Health. 

During  the  symposium,  three  presentations  described 
initiatives  that  exemplify  the  principles  and  components  of 
the  Ecology  of  Outcomes.  These  included  development  of 
the  statewide  information  system  for  the  Texas  Children's 
Mental  Health  Plan;  lessons  learned  from  the 
Pennsylvania's  SumOne  for  Kids,  developed  by  Pressley 
Ridge  School;  and  California's  system  of  care  approach  to 
outcome  tracking.  Summaries  of  the  Texas  and  Pensylvania 
activities  are  presented  in  this  volume. 

The  Ecology  of  Outcomes:  System  Accountability  in 
Children's  Mental  Health 

Mario  Hernandez,  Ph.D.  Sharon  Hodges,  M.B.A. 

Introduction 

A fundamental  reason  for  tracking  outcomes  in  applied 
service  settings  is  to  determine  whether  the  person  receiving 
services  benefits  in  an  observable  manner  as  a result  of  the 
services  provided.  For  the  purposes  of  this  summary, 
outcomes  are  defined  as  the  results  or  the  impact  of  services 
provided  to  children  and  their  families.  Furthermore, 
outcome  accountability  can  be  defined  as  the  systems  of 
care's  responsibility  for  accomplishing  publicly  articulated 
goals  of  service  provision,  as  measured  through  accurate 
monitoring  over  time. 

The  purpose  of  this  summary  is  to  present  a framework  that 
can  serve  as  the  foundation  for  utilization  of  outcome 
information  to  provide  opportunities  for  learning  and 
self-correction.  The  four  sets  of  components  of  the  Ecology 
of  Outcomes  framework  include  1)  Principles  for  Outcomes 
Accountability;  2)  Prerequisites  and  Building  Blocks  for 
building  outcome  accountability;  3)  guidelines  for 
Implementing  an  Outcome  Information  System;  and  4)  an 
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approach  for  Utilizing  the  Results.  The  Ecology 
framework's  emphasis  on  outcome  information  as  a 
resource  for  use  in  decision  making  is  evident  in  the 
Principles  for  Outcome  Accountability.  The  next  sections  of 
this  summary  present  the  principles  for  outcome 
accountability,  and  then  discusses  each  component  of  the 
framework. 

Principles  for  Outcome  Accountability 

Stroul  and  Friedman  (1986)  recognized  that  although  the 
components  and  organizational  structure  of  a child-serving 
system  might  vary  from  state  to  state  or  community  to 
community,  the  development  and  implementation  of  a 
system  of  care  should  be  guided  by  a set  of  values  and 
principles.  Similarly,  there  are  principles  central  to  the 
successful  development  and  integration  of  outcome 
information  into  the  planning  and  delivery  of  services  that 
transcend  the  variability  and  unique  characteristics  of  any 
child-serving  system's  components  and  organizational 
structure.  Ten  guiding  principles  have  been  identified  for 
the  development  and  utilization  of  outcome  information  in 
systems  of  care  (Hernandez  & Hodges,  1996).  These 
principles,  listed  below,  are  central  to  the  design  and 
operation  of  an  outcome  accountability  approach  because 
they  specify  what  drives  and  shapes  the  development  and 
implementation  of  the  framework. 

1. Outcome  information  cannot  be  collected  in  isolation  of 
information  about  who  is  served  and  what  services  are 
offered. 

2.  Outcome  information  should  be  used  to  improve  service 
planning  and  delivery 

3. Outcome  information  should  be  relevant  and  accessible  to 
key  stakeholders  in  the  system  of  care. 

4. The  application  and  use  of  outcome  information  should  be 
informed  by  the  available  research  base. 

5. Outcome  information  should  support  culturally  competent 
decision  making  in  service  planning  and  delivery. 

6.  Key  stakeholders  should  be  involved  in  identifying  and 
selecting  outcomes  to  be  measured. 

7.  Data  elements  for  the  outcome  measurements  should  be 
clearly  defined. 

8. Outcome  information  should  be  useful  to  managers, 
administrators,  and  direct  service  providers. 

9.  The  process  for  collecting,  analyzing  and  communicating 
outcome  results  should  be  timely  and  occur  on  a predictable 
schedule. 
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10.  Outcome  information  should  provide  the  opportunity 
for  corrective  action. 

Prerequisites  and  Building  Blocks 

Prerequisites  and  Building  Blocks,  discussed  below,  are  two 
essential  components  of  successful  implementation  of  an 
outcome  accountability  approach.  They  should  be  seen 
together  as  laying  a foimdation  on  which  accountability  can 
be  built  and  thrive  tsee  Figure  IT 

The  purpose  of  the  Prerequisite  component  is  to  assess  a 
service  system's  level  of  commitment  to  building  an 
outcome  accountability  system.  In  essence,  the  Prerequisite 
phase  should  be  used  to  determine  if  there  is  enough 
momentum  and  motivation  to  establish  and  sustain  an 
outcome  system.  Two  aspects,  leadership  and  political 
climate  are  key  to  the  Prerequisite  phase  of  the  Ecology  of 
Outcomes  framework. 

The  Building  Blocks  component  represents  the 
development  of  a plan  for  building  and  implementing  an 
outcome  system.  The  primary  tasks  for  this  component 
involve  clarifying  the  reasons  for  developing  an  outcome 
system,  describing  what  needs  to  be  accomplished  in  order 
to  implement  accountability,  and  determining  baseline 
levels  to  establish  the  current  status  of  relevant  data.  The 
primary  aspects  of  the  Building  Blocks  phase  of 
development  are  (a)  establishing  a process  for  involving 
stakeholders;  (b)  clarifying  the  language  of  outcome  and 
accoimtability;  (c)  assessing  current  capacity  for  building  a 
system  of  outcome  accountability;  and  (d)  planning  for 
implementation. 

The  significant  challenge  inherent  in  both  components  is 
building  a shared  vision  among  stakeholders  about  what 
shape  the  accountability  approach  will  take  when  complete. 
While  this  can  be  a formidable  task,  if  not  addressed,  it  can 
lead  to  a breakdown  in  the  development  process 
(Meadowcroft,  Pierce,  and  Beck,  1 994). 

When  successfully  completed,  activities  described  by  the 
aspects  of  Building  Blocks  and  Prerequisites  components 
yield  consensus  among  key  stakeholders  about  who  the 
system  hopes  to  serve,  what  services  are  expected  to  be 
provided,  and  what  outcomes  the  system  hopes  to  produce. 
All  three  elements  of  consensus  are  critical  to  a fully 
functioning  and  useful  accountability  approach  (Usher, 
1993a,b).  If  only  outcomes  were  tracked  and  reported,  it 
would  be  impossible  to  use  the  information  to  improve 
service  delivery;  that  is  information  and  data  without 
context,  purpose,  and  interpretation  is  useless. 

The  Outcome  Information  Components 

The  Ecology  framework  maintains  that  outcome 
information  cannot  be  used  in  isolation  of  information 
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about  who  is  being  served  and  what  services  are  being 
provided.  From  this  perspective  there  are  three  components 
to  the  outcome  information  framework:  1)  populations 
targeted  for  services;  and  2)  what  services  are  provided;  and 
3)  information  about  what  outcomes  have  been  achieved. 

With  respect  to  answering  questions  of  who  is  being  served, 
the  Ecology  framework  suggests  that  tests  that  two  broad 
categories  of  information  about  children  and  families  will 
be  useful  in  service  planning  and  delivery:  1)  information 
about  children  and  families  that  makes  it  possible  to 
determine  whether  the  system  serves  the  children  and 
families  it  intended  to  serve;  and  2)  other  information  about 
child  and  family  characteristics  that  may  influence  the 
system's  outcomes. 

Fundamentally,  service  providers  and  other  stakeholders 
need  to  know  that  the  populations  they  intend  to  serve  are, 
in  fact,  being  served.  A system  which  fails  to  serve  its 
intended  population  cannot  accurately  assess  its  outcomes. 
In  addition  to  information  about  target  population 
characteristics,  information  about  other  child  and  family 
characteristics  can  be  useful  in  the  interpretation  of  what 
may  have  influenced  an  achieved  outcome.  Burns  (in  press) 
provides  a list  of  suggested  child  and  family  characteristics 
which  may  influence  outcomes.  These  include  risk  factors 
such  as  poverty,  family  history  of  mental  illness;  illness 
severity,  chronicity  and  co-morbidity;  family  strengths  and 
tolerance  of  stress;  social  support;  family  member's  case 
management  skills;  and  treatment  adherence  by  family 
members  and  therapists.  Combining  a limited  number  of 
carefully  chosen  child  and  family  characteristics  with 
information  about  whether  target  populations  have  been 
served  can  greatly  enhance  a system's  ability  to  interpret  its 
outcomes  more  confidently. 

In  considering  how  to  describe  services  that  are  being 
provided,  the  Ecology  framework  suggests  that  four  aspects 
are  useful  for  tracking  and  monitoring  services,  as  well  as 
interpreting  future  outcome  information  (Hernandez  & 
Goldman,  1996).  These  are:  1)  intensity,  frequency  and 
duration  of  services;  2)  location  of  services;  3)  variety  and 
sequencing  of  services;  and,  4)  integrity  of  services. 

This  approach  requires  that  planners  in  organizations 
articulate,  in  operational  terms,  what  services  they  expect  to 
offer.  This  operationalization  and  tracking  of  service 
aspects  gives  service  systems  personnel  confidence  that  the 
results  of  their  efforts  can  be  plausibly  related  to  the  type  of 
services  they  provide  (Dym,  1996).  The  Ecology 
framework  sees  outcome  information  as  a measure  of  what 
the  system  has  accomplished.  It  should  be  emphasized  that 
the  Ecology  framework  stresses  the  use  of  outcomes  in  the 
context  of  managerial  needs  rather  than  for  purposes  of 
generalizability  and  application  to  larger  social  contexts.  It 
is  necessary  to  establish  specific  criteria  for  the  selection  of 
outcomes  to  be  measured.  The  Ecology  framework  uses  a 
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series  of  questions,  shown  below,  to  guide  the  selection  of 
outcomes. 

• Is  the  outcome  information  useful  to  managers  and 
administrators? 

• Is  the  outcome  information  useful  to  front-line 
workers? 

• Do  the  outcome  results  provide  opportunity  for 
corrective  action? 

• Do  the  outcome  results  support  the  achievement  of 
cultural  competence? 

Once  outcome  domains  have  been  selected,  a second  layer 
of  decision  making  has  to  occur  in  order  to  select  the 
indicators.  That  is,  decisions  must  be  made  as  to  what 
indicators  will  be  used  to  measure  the  outcomes  and  what 
criteria  should  be  applied  in  making  the  selection.  The 
Ecology  framework  offers  several  questions  which  may  be 
useful  in  selecting  the  indicators.  These  are  shown  below: 

• Does  the  indicator  adequately  represent  the  status  of 
an  outcome? 

• Is  the  indicator  easily  measured? 

• Is  the  process  of  data  collection  and  reporting  realistic 
and  sustainable? 

• Does  the  indicator  provide  valid  and  reliable 
information  about  the  outcome? 

Utilizing  the  Results 

The  Ecology  framework  assumes  that  using  outcome 
information  is  a process,  not  an  event  (Bums,  in  press). 

Two  primary  elements  in  the  process  of  utilizing  the  results 
rest  on  this  assumption.  These  elements  are  1)  the  process 
of  interpreting  the  outcome  information,  and  2)  action 
decisions  made  as  a result  of  what  has  been  learned.  Figure 
2 illustrates  that  output  is  produced  by  the  child-serving 
system  in  the  form  of  system  information  regarding  who  the 
system  has  served,  what  services  have  been  provided,  and 
what  outcomes  have  been  produced. 

This  interpretive  process  requires  returning  to  the  work 
generated  in  the  Building  Blocks  component  and  measuring 
outcomes  against  goals  that  were  developed  for  the  service 
system.  The  interrelationships  among  who  was  served,  what 
services  were  provided,  and  what  outcomes  have  resulted 
must  be  considered  in  the  interpretive  process.  Baseline 
information  about  all  elements  at  the  beginning  of  the 
measurement  period  becomes  the  reference  for 
understanding  the  meaning  of  the  information  and  results. 

The  focus  of  the  action  step  is  on  modifying  service 
planning  and  delivery,  as  needed,  based  on  an  assessment  of 
the  status  of  the  results.  This  use  of  the  interpretive  process 
to  inform  a decision  to  either  change  or  not  change  aspects 
of  service  planning  and  delivery  is  best  understood  as  a 
process  of  working  toward  improved  results  rather  than  an 
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end  result.  Rather  than  a static,  one-time  process,  a system 
of  utilizing  the  results  should  be  embedded  into  day-to-day 
management. 

Implications  for  Children's  Mental  Health 

The  shift  in  interest  toward  results-based  accountability 
raises  hopes  that  mental  health  systems  will  respond  more 
flexibly  to  those  they  serve,  that  public  faith  in  the  ability  of 
human  service  institutions  to  accomplish  their  intended 
purposes  will  be  restored,  and  that  communities  will  be 
better  able  to  plan  their  support  of  children  and  families 
(Schorr  et  al.  1 994).  A System  of  Care  for  Severely 
Disturbed  Children  and  Youth  (Stroul  & Friedman,  1986) 
more  clearly  defined  the  concept  of  system  of  care  and 
provided  guidance  in  how  to  build  systems  that  would 
allow  children  to  receive  services  while  remaining  at  home 
and  in  their  communities.  We  believe  the  Ecology  of 
Outcomes  framework  will  both  complement  and  expand  the 
systems  of  care  concept  by  helping  policy  makers  and 
administrators  establish  strategies  to  build  outcome 
information  systems  and  incorporate  outcome  information 
into  decisions  that  impact  the  planning  and  delivery  of 
services  to  children  and  their  families. 
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The  Development  of  a Continuous  Evaluation  System 
for  the  Texas  Children's  Mental  Health  Plan:  Building 
an  Evaluation-Stakeholder  Feedback  Loop 

Lawrence  W.  Rouse,  Ph.D. 

Introduction 

This  summary  focuses  on  the  development  of  an  ongoing 
evaluation  system  associated  with  the  Texas  Children's 
Mental  Health  Plan  (TCMHP)  and  the  steps  taken  to  build  a 
feedback  loop  between  evaluators  and  stakeholders.  The 
Texas  Children's  Mental  Health  Plan  is  an  interagency 
effort  to  develop  a continuum  of  community-based  mental 
health  services  for  children,  adolescents,  and  their  families 
based  on  the  federal  Children  and  Adolescent  Services 
System  Program  Model  (CASSP).  The  ongoing  evaluation 
was  developed  in  order  to  provide  children's  mental  health 
administrators  at  the  state  and  local  level  with  information 
about  children's  mental  health  services  and  demonstrate 
accountability  to  the  consumers  and  funding  sources. 

The  purpose  of  this  summary  is  to:  (a)  explain  how 
outcomes  are  important  to  the  basic  philosophy  of  TCMHP; 
(b)  describe  the  system  for  providing  information  to 
st^eholders  (e.g.,  consumers,  service  providers,  program 
managers,  advocates,  and  legislators);  and  (c)  describe  the 
interactions  that  have  taken  place  between  evaluators  and 
various  TCMHP  stakeholders  in  creating  an  ongoing  stream 
of  evaluation  information  for  decision  making. 
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The  Texas  Children 's  Mental  Health  Plan 

In  1992,  the  Texas  legislature  appropriated  monies  to  the 
Texas  Department  of  Mental  Health  and  Mental  Retardation 
(TXMHMR)  for  the  implementation  of  the  Texas  Children's 
Mental  Health  Plan.  The  prim^  goal  of  the  TCMHP  is  to 
develop  and  implement  a public  community-based  mental 
health  system  for  children,  adolescents,  and  their  families 
through  the  coordination  of  resources  of  all  the  state 
child-serving  agencies.  An  essential  feature  of  the  TCMHP 
is  the  participation,  at  the  state  and  local  level,  of  the 
child-serving  agencies,  advocates,  and  consumers  in 
management  teams  with  the  express  purpose  of  making 
collaborative  decisions  about  TCMHP  activities.  TCMHP 
services  are  organized  into  three  components:  (a)  "core" 
mental  health  services,  (b)  services  to  children  referred  from 
the  juvenile  justice  system,  and  (c)  early  intervention  and 
prevention  services.  During  FY  '95,  a total  of  26,000 
children  were  served  through  TCMHP  services. 

Another  essential  feature  of  the  plan  is  a list  of  outcomes  to 
be  measured  for  each  of  the  components  of  the  TCMHP. 

The  outcomes  were  written  into  the  plan  from  the  very 
beginning  to  assure  the  stakeholders,  as  well  as  the  state 
legislature,  that  the  effectiveness  of  the  services  were  being 
measured  and  that  decisions  about  the  TCMHP  were  being 
assisted  by  evaluation  data.  In  addition,  TXMHMR  has 
been  committed  to  the  implementation  of  the  principles  of 
Total  Quality  Management  (TQM)  as  a work  philosophy. 
One  of  the  hallmarks  of  the  TQM  approach  is  the 
measurement  of  work  activities,  using  this  information  in 
modifying  work  processes  to  increase  productivity  and 
effectiveness.  Finally,  in  its  shift  towards  a managed  care 
organizational  mode,  TXMHMR  has  recently  begun 
incorporating  "outcomes  to  be  attained"  in  its  contracts  with 
each  of  the  community  service  sites.  Therefore,  the 
collection  and  dissemination  of  evaluation  information  to 
TXMHMR  and  community  site  managers  is  central  to  the 
continual  development  of  the  TCMHP. 

Description  of  the  Evaluation 

The  development  of  a continuous  evaluation  system  for  the 
TCMHP  was  characterized  by  three  stages  that  began  in 
1990.  The  first  stage  was  a summative  evaluation  of  the 
impact  of  one  main  service  type  on  child  and  family 
functioning  at  five  mental  health  centers.  In  1992,  the 
evaluation  was  extended  to  16  sites  and  included  the 
ongoing  evaluation  of  14  service  types.  By  1994,  the 
evaluation  was  implemented  at  all  45  community  mental 
health  authorities  across  the  state.  The  evaluation  currently 
involves  all  children  served  in  the  public  mental  health 
system  and  1 8 service  types. 

The  evaluation  is  managed  by  a committee  of  professional 
evaluation  personnel  representing  the  nine  state  agencies 
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which  participate  in  the  TCMHP,  representatives  from 
consumer  advocate  groups,  program  directors  from  the 
sites,  and  a parent  representing  the  viewpoint  of  the 
consumers. 

The  basic  evaluation  design  is  characterized  by 
pretreatment,  post-treatment,  and  follow-up  measurements 
of  consumer  demographics,  history,  outcomes,  and 
satisfaction  with  services.  Outcomes  of  services  include 
general  psychological  functioning,  behavioral-emotional 
functioning,  out-of-home  placement  rates,  and 
social-community  functioning.  A multi-method/multi-rater 
approach  to  measurement  is  employed  to  collect 
information  from  the  service  providers,  children,  parents, 
and  collateral  providers  using  interviews,  checklists,  and 
rating  scales.  Data  is  collected  primarily  by  the  service 
providers  at  pre-and  post-treatment.  Follow-up  data  is 
collected  by  the  TXMHMR  research  office. 

Communicating  evaluation  results  has  been  accomplished 
through  the  establishment  of  the  Quarterly  Service  Report, 
Quarterly  Report  Review,  and  special  reports.  The 
Quarterly  Service  Report  provides  a summary  of  key 
indicators  in  the  areas  of  numbers  served,  demographics, 
and  outcome  measures.  The  Quarterly  Report  Review  is 
used  as  a vehicle  for  a discussion  of  the  figures  on  the 
Quarterly  Service  Report  and  also  provides  an  opportunity 
for  the  publishing  of  other  evaluation  results  that  may  be 
relevant.  Special  reports  are  also  produced  for  TXMHMR 
center  managers  and  other  stakeholders  as  requested.  A 
catalogue  of  special  reports  is  maintained  and  available  for 
reference.  Taken  together,  these  publications  are  seen  as  an 
essential  tool  in  facilitating  a most  important  set  of 
evaluation-stakeholder  transactions  (i.e.,  a feedback  loop). 

Evaluator-Stakeholder  Feedback  Loop 

The  feedback  loop  for  the  TCMHP  is  characterized  by  at 
least  four  major  activities  that  are  constantly  being 
reiterated.  The  loop  begins  as  stakeholders  raise  questions 
about  the  program.  Evaluators  collect  and  analyze  data  in 
response  to  these  questions.  Evaluators  then  collaborate 
with  stakeholders  in  using  results,  and  stakeholders  use 
conclusions  about  results  in  making  decisions,  which  lead 
to  new  questions,  beginning  the  cycle  again. 

Within  each  of  these  activities,  interactions  between 
evaluators  and  stakeholders  provide  inertia  for  the 
reiteration  of  the  feedback  loop.  Throughout  the 
development  of  the  TCMHP  evaluation,  these  interactions 
have  contributed  to  modifications  in  major  activities  in 
order  to  meet  the  needs  of  both  the  evaluators  and  the 
stakeholders. 

Stakeholders  Raise  Initial  Questions 

The  provision  of  guidelines  for  evaluation  of  TCMHP 
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outcomes  created  an  important  dialogue  between  the 
authors  of  the  plan  and  the  evaluation  committee.  These 
interactions  established  evaluation  as  part  of  the  plan  from 
the  beginning  and  also  helped  formulate  and  clarify  the 
initial  evaluation  questions.  During  this  process,  the 
evaluation  committee  voted  on  specific  evaluation 
questions,  methods  and  measures.  As  the  evaluation 
progressed,  new  members  were  added  to  the  evaluation 
committee  to  provide  additional  guidance. 

Collection  and  Analysis  of  Data 

Perhaps  the  most  attention  has  been  paid  to  the  process  for 
the  collection  of  information  and  the  analysis  of  data.  The 
original  evaluation  design  included  the  measurement  of 
outcomes  for  each  service  type  resulting  in  several 
assessments  for  each  child.  In  order  to  make  the  evaluation 
an  integral  part  of  service  activity,  the  design  needed  to  be 
simplified.  Through  meetings  with  groups  of  service 
providers  and  surveys  of  program  directors,  it  was 
determined  that  too  much  effort  was  needed  to  collect 
multiple  assessments,  and  it  was  unlikely  that  clients  could 
distinguish  between  the  different  service  types.  This 
impression  was  confirmed  through  flowcharting  of  the 
evaluation  process  conducted  by  the  evaluation  committee. 
When  pictures  of  the  processes  and  transactions  involved  in 
the  evaluation  were  analyzed,  a dramatic  picture  of  a 
complicated  and  cumbersome  flow  of  activity  was  revealed. 
Consequently,  a less  complex  evaluation  design  was  created 
based  on  an  episode  of  care. 

In  order  to  reduce  the  burden  of  data  collection  on  the 
service  providers  and  provide  managers  with  a minimum 
set  of  key  outcome  indicators,  the  measures  were  revised. 
Program  directors  were  surveyed  as  to  which  measures 
were  most  meaningful  to  them.  They  suggested  use  of  the 
CBCL  and  satisfaction  forms  and  elimination  of  the 
provider  completed  pre-and  post-  treatment  assessment 
forms.  Concurrently,  the  initial  measures  were  reviewed  for 
the  frequency  of  use  in  data  analyses  and  their  psychometric 
properties.  It  was  discovered  that  service  providers'  ratings 
of  treatment  plan  completion  were  infrequently  used,  and 
subscales  of  provider  completed  pre-  and  post-  assessment 
measures  had  mixed  psychometric  properties.  Satisfaction 
forms,  however,  showed  good  reliability  and  validity. 

A major  effort  also  was  made  to  integrate  the  evaluation 
into  preexisting  processes  to  minimize  paperwork  for 
service  providers  while  supporting  efforts  to  document 
compliance  with  standards.  The  evaluation  was  dove-tailed 
with  the  Department's  efforts  to  meet  defined  mental  health 
community  standards  such  as  continuity  of  care,  service 
type  descriptions,  outcome  standards,  and  client  assessment 
and  treatment  plan  requirements.  Additionally,  the 
evaluation  utilized  the  state-wide  client  registration  and 
assignment  data  base,  thus  automating  many  of  the  data 
collection  procedures. 
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Technical  assistance  to  the  field  was  viewed  as  an  essential 
part  of  implementation  of  the  evaluation.  As  a first  step,  the 
evaluation  committee  felt  that  program  directors  and  service 
providers  would  benefit  from  information  on  evaluation, 
data  management,  and  how  the  state  and  local  computer 
systems  worked.  Regional  trainings  were  implemented  to 
introduce  the  evaluation,  followed  by  on-site  evaluation  and 
telephone  training.  Initially,  training  was  provided  to  serve 
providers  and  program  directors.  Later,  staff  from  medical 
records  and  information  services  were  included.  High 
ratings  in  five  of  six  training  sessions  suggested  that 
participants  found  the  sessions  to  be  informative  and 
helpful  in  implementing  work  tasks. 

As  the  evaluation  proceeded,  quality  control  measures  were 
implemented  so  the  data  used  in  analyses  would  be 
credible.  The  computerized  data  collection  system  was 
edited  to  force  completion  of  data  elements.  Additionally, 
manual  editing  of  data  forms  was  performed  and  feedback 
has  been  given  back  to  the  centers.  To  further  assess  the 
accuracy  of  data,  a pilot  has  been  implemented  to  dovetail 
with  SQA  audits. 

Special  emphasis  was  placed  on  empowering  staff  to  make 
their  own  decisions  about  how  to  implement  the  evaluation 
at  their  own  center.  Efforts  were  made  to  communicate  to 
the  staff  that  the  evaluation  instructions  were  often 
guidelines  and  they  needed  to  operationalize  and  consider 
local  needs.  For  example,  on  site  decision  issues  included 
data  management  and  the  coordination  of  data  collection 
tasks  with  naturally  occurring  clinical  activities.  As  a result, 
staff  included  pre-  and  post-  assessments  in  intake  and 
discharge  activities,  and  integrated  registration  and  history 
information  into  center-specific  computer  system  data 
entry. 

Collaborating  with  Stakeholders 

Evaluators  must  collaborate  with  the  stakeholders  in  using 
the  results  of  evaluation.  Managers  and  service  providers 
need  education  to  read  and  interpret  data  from  clinical  and 
administrative  viewpoints.  A service  report  was  created  to 
report  data  back  to  stakeholders  on  a regular  basis, 
presenting  data  in  a tabular  fashion.  Later,  a text  was 
produced  to  discuss  the  data,  to  provide  suggestions  on 
what  the  data  meant,  as  well  as  how  it  could  be  used  to 
monitor  programs.  Finally,  graphic  representations  of  trends 
were  provided. 

Surveys  were  then  implemented  to  determine  if  the 
presentation  of  evaluation  information  was  adequate  for  the 
needs  of  TXMHMR  managers  and  program  directors. 
Results  suggested  that  the  service  report  and  accompanying 
text. met  their  needs.  Additionally,  informants  indicated  that 
preferred  elements  of  evaluation  reports  included  data  about 
outcomes,  graphics,  and  information  in  a bullet  format. 
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Data  about  consumer  satisfaction  was  a less  frequent 
preference.  In  response  to  this  feedback,  a new  format  for 
the  service  reports  has  been  developed  which  features  more 
outcome  information.  In  addition,  the  increase  over  time  of 
requests  for  special  data  runs  demonstrates  that  stakeholders 
are  finding  value  in  the  evaluation  process. 

Evaluation  Results  and  Program  Decisions 

If  evaluation  data  are  not  used  in  decision  making,  then  the 
evaluation  has  not  reached  its  intended  goal.  Up  to  this 
point,  the  emphasis  of  the  TCMHP  has  been  placed  on 
putting  an  evaluation  system  in  place.  Future  activity  must 
measure  the  extent  to  which  the  evaluation  data  are  being 
used  to  assist  in  making  program  decisions. 

Currently,  the  only  measure  of  the  use  of  evaluation  data  is 
anecdotal  information  such  as  program  directors'  reports 
that  the  data  have  been  helpful  in  particular  instances. 
However,  there  has  been  increased  contact  from  program 
directors  and  TXMHMR  managers  to  request  information, 
and  TXMHMR  managers  have  requested  that  special 
reports  be  prepared  for  legislative  aids  to  support  the 
funding  process. 

Discussion 

The  implementation  of  an  ongoing  evaluation  system  for 
the  TCMHP  has  been  a developmental  process  of 
implementation  and  revision,  obtaining  feedback  from 
external  and  internal  customers  and  revision  again. 
Establishing  interactions  with  the  stakeholders  to  solve  the 
problems  of  implementing  a continuous  evaluation  seems 
best  accomplished  by  starting  small  and  expanding  once 
major  issues  are  identified. 

Clearly,  the  development  of  an  evaluation  process  has  much 
to  offer  to  the  system  to  be  evaluated.  In  the  present 
situation,  the  TCMHP  evaluation  has  helped  the  program 
directors  define  the  services  they  are  offering  and  interpret 
specific  aspects  of  the  community  standards.  It  has  helped 
establish  outcomes  to  be  included  in  contract  negotiation 
and  monitoring  and  changed  the  statewide  client  data 
system  to  be  more  relevant  to  children  and  families  served 
in  the  system. 

Taken  together,  these  experiences  and  lessons  learned  seem 
to  indicate  that  once  the  initial  turn  of  the  feedback  loop  is 
accomplished,  then  further  reiterations  of  the  loop  are  more 
easily  attained. 

SumOne  for  Kids:  Measuring  and  Improving  Results  in 
Services  for  Children  and  Families 

Pamela  Meadowcroft,  Ph.D. 

Introduction 
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SumOne  for  Kids  is  a multi-agency  outcome  monitoring 
system  developed  through  a collaboration  between  The 
Pressley  Ridge  Center  for  Research  and  Public  Policy  and 
3 1 private,  nonprofit  child  serving  agencies  in 
Pennsylvania.  The  original  goal  was  to  create  a system  with 
all  technology  supports,  including  functional  software,  all 
measurement  tools  and  other  data  collection  devices, 
training  and  audit  services  that  was  (a)  low  cost,  (b)  could 
be  used  at  the  agency  or  program  level  for  program 
improvement,  (c)  would  form  the  basis  of  a central  database 
that  would  produce  reports  useful  to  policy  makers  and 
providers,  and  (d)  would  answer  the  questions:  Who  are  the 
children  and  families  we  serve?  What  services  do  they 
receive  at  what  cost?  What  is  the  impact  of  these  services 
on  their  lives?  To  create  the  central  database,  all 
participating  agencies  would  agree  to  upload  a key  set  of 
data  elements  from  which  aggregate  reports  would  be 
generated  for  comparison  purposes.  In  this  way,  agencies 
could  compare  their  own  results  with  the  combination  of  all 
agencies  and  aggregate  reports  could  better  inform 
policymakers  of  the  results  of  children's  services. 

The  initial  pilot  provider  agencies  served  over  5000 
children  and  families  every  day  from  all  of  the  major 
child-serving  systems  including  mental  health,  child 
welfare,  juvenile  justice,  and  special  education.  The  types 
of  services  they  provided  also  represent  the  full  array  of 
services  to  children  and  families,  including  in-home  and 
family  preservation  services,  adoption,  day  treatment  and 
partial  hospital  programs,  foster  family  care,  therapeutic 
foster  care,  group  homes,  and  residential  treatment. 
Therefore,  the  outcome  evaluation  system  was  designed  to 
be  useful  for  all  children's  services  and  do-able  by  provider 
agency  staff. 

The  following  are  some  of  the  lessons  learned  and  values 
developed  from  the  mistakes  we  made  and  the  barriers  we 
experienced  in  creating  SumOne  for  Kids. 

Lesson  1 

The  first  rule  of  comedy,  politics,  and  sex,  and  now 
outcome  measurement,  is  that  timing  is  everything.  The 
original  start-up  for  SumOne  for  Kids  six  years  ago  was 
painfully  slow.  Funding  was  not  immediately  available 
since  the  Foundation  community  at  that  time  did  not  view 
outcome  evaluation  to  be  an  urgent  priority.  Nor  did  the 
payers  of  children's  services  view  measurement  of 
outcomes  as  a necessity.  For  example,  for  eight  years  the 
results  of  the  direct  services  programs  at  Pressley  Ridge  had 
been  evaluated  by  contacting  the  few  hundred  children  and 
families  whose  services  had  been  completed  the  year 
before.  This  follow-up  evaluation  served  as  the  prototype 
for  SumOne  for  Kids  and  provided  Pressley  Ridge 
management,  clinicians,  and  the  board  of  trustees  with  a 
way  of  focusing  priorities  for  each  year's  program 
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development  activities.  However,  not  once  in  the  eight 
years  did  the  agency's  referral  of  funding  sources  ask  for  or 
use  the  outcome  results.  Such  disinterest  has  dramatically 
changed  in  the  last  two  years.  The  pace  and  subsequent 
funding  and  interest  in  outcome  evaluation  in  general,  and 
SumOne  for  Kids  in  particular,  has  exploded. 

Lesson  2 

Build  on  what  the  users  of  the  outcome  monitoring  system 
are  already  doing.  SumOne  for  Kids  staff  take  participating 
agencies  through  a design  process  that  builds  on  what  they 
are  already  collecting.  The  original  pilot  group  of  agencies 
helped  determine  all  of  the  data  elements  that  fully  describe 
the  children  and  families  served  and  the  types  of  services 
received.  In  replications,  this  same  customizing  process  is 
used  to  ensure  that  participating  groups  have  input  into  the 
data  that  is  required  by  the  outcome  system.  The  pilot 
agencies  also  fully  tested  the  outcome  interviews  that  were 
developed,  the  forms  that  were  used  to  catch  the  data 
elements,  and  the  initial  software. 

Lesson  3 

Make  the  data  useful  and  easy  to  get.  Most  provider 
agencies  already  collect  lots  of  data.  But  the  data  are  only 
useful  when  (a)  it  is  in  a readily  retrievable,  readable  form 
(i.e.  attractive,  easy  to  read  reports);  and,  (b)  when  it  can  be 
compared  to  benchmarks  so  the  results  have  meaning. 
SumOne  for  Kids  built  into  the  software  standardized 
reports  that  the  pilot  agencies  tell  us  are  essential  for  the 
day  to  day  operating  of  their  services.  The  standard  reports 
were  designed  and  tested  to  be  readable  by  nonresearch 
staff;  hence,  agencies  can  use  the  reports  without  research 
staff. 

Lesson  4 

There  must  be  a strong  incentive  for  agencies/systems  to 
become  accountable.  The  impetus  for  SumOne  for  Kids 
came  in  1989  when  it  began  to  look  as  if  a system  of 
accountability  might  be  imposed  on  Pennsylvania  providers 
by  the  legislature.  The  provider  agencies  decided  to  be 
pro-active,  develop  their  own  system,  and  then  turn  it  over 
to  the  state  for  use  as  a state-imposed  monitoring  system 
(bottom-up/top  down  approach).  While  SumOne  for  Kids  is 
based  on  self-evaluation  assumptions  in  which  agencies 
want  to  know  how  effective  their  services  are,  outcome 
monitoring  will  require  mandates  in  some  form  to  insure  the 
level  of  commitment  required  of  agencies  to  produce 
accurate,  timely  data.  Only  recently  has  Pennsylvania  begun 
to  mandate  outcome  evaluation  in  different  forms.  For 
example,  in  Allegheny  County,  outcome  evaluation  is  now 
part  of  the  contracting  process  with  all  Children  and  Youth 
Services  providers. 

Lesson  5 


134 


Get  political  support  early  on  and  throughout  the 
development  and  implementation  of  any  outcome 
monitoring  system.  An  outcome  monitoring  system  may  be 
technically  valid  and  sophisticated,  but  not  at  all  useful  if  it 
does  not  have  widespread  political  support.  When  SumOne 
for  Kids  began,  a group  of  bureaucrats,  policy  makers, 
educators,  researchers,  and  practitioners  was  assembled  for 
the  purpose  of  articulating  a vision  of  what  such  a system 
might  look  like  and  how  it  might  operate.  Everyone  present 
gave  high  praise  to  the  concept,  pledged  the  full  support  of 
their  offices,  and  asked  to  be  kept  informed  of  project 
status.  The  same  process  has  been  used  even  more 
successfully  in  Maryland,  where  there  are  no  regulatory 
requirements  for  program  evaluation  in  residential  services. 

Lesson  6 

"Outcomes"  lack  clear  definition.  Although  outcomes  are 
now  talked  about  by  persons  at  a variety  of  levels  and  in 
many  kinds  of  systems,  there  is  little  agreement  on  what 
they  are.  Practitioners  are  interested  in  clinical  outcomes 
that  relate  to  the  child's  treatment  plan.  Program  managers 
and  states  are  more  often  interested  in  process  measures, 
such  as  numbers  of  service  units  provided,  or  number  of 
children  served,  and  will  consider  these  the  "outcomes"  of 
importance.  Project  staff,  however,  took  a stand  early-on 
that  we  would  look  at  socially  significant  impacts  of  the 
services  provided  on  the  lives  of  children  and  families 
served.  Such  functional  outcomes  can  appeal  to  the 
practitioner,  program  manager,  as  well  as  state  level 
policy-makers. 

To  assess  functional  outcomes  requires  a commitment  to 
looking  at  the  results  of  services  after  receipt  of  these 
services.  Project  staff  found  numerous  barriers  to  this  view 
of  outcome  measurement.  Many  providers  believe  their 
accountability  ends  when  the  child  leaves  their  program. 
Others  feel  that  post-discharge  results  can  be  useful  in 
determining  program  change,  but  there  is  no  agreement  on 
how  long  after  discharge  that  responsibility  lasts.  The 
project  decided  to  ask  a large  group  of  varied  children's 
services  stakeholders  to  define  outcomes  for  us,  thus 
avoiding  having  to  debate  the  outcomes  definition  with 
evaluation  experts  or  anxious  providers. 

Lesson  7 

Involve  stakeholders  in  key  aspects  of  developing  outcome 
measures.  This  lesson,  as  all  the  others,  was  learned  over 
and  over  again.  A large-scale  social  validation  survey  of  | 
over  700  Pennsylvania  stakeholders  of  children's  services 
defined  the  outcome  indicators  for  us.  (This  survey  was  | 
reported  at  the  Research  Conference  in  1992.)  The  survey 
asked  respondents  to  indicate  how  important  various  issues 
were  to  them  and  how  satisfied  they  were  with  the  services 
available  in  their  community  to  address  those  issues.  With  a 1 
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90%  response  rate,  project  staff  felt  they  had  solid  evidence 
of  what  issues  should  be  put  forward  as  the  most  important 
ones  to  measure,  which  were: 

• stability  and  restrictiveness  of  children's  living 
environments; 

• use  of  drugs  and  alcohol; 

• school  attendance  and  graduation; 

• employment  and  job  readiness;  and 

• protection  from  harm. 

The  3 1 pilot  agencies  provided  SumOne  for  Kids  staff  with 
the  sites  to  test  out  the  various  tools  that  were  designed  to 
measure  the  above  outcome  indicators.  A similar 
stakeholder  survey  was  conducted  in  Maryland  with  over 
1000  participants.  The  results  bore  a striking  similarity  to 
that  of  the  Pennsylvania  survey. 

Lesson  8 

Generate  products  quickly  and  keep  the  momentum  active 
or  participants  will  lose  interest,  at  best.  A certain 
momentum  must  be  achieved  and  maintained  to  keep 
participants  involved,  and  there  needs  to  be  regular 
communication  between  project  staff  and  its  participants. 
Newsletters,  reports  on  results  of  each  project  step,  getting 
part  of  the  data  system  operating  and  producing  results  right 
away  are  some  of  the  tactics  project  staff  learned  along  the 
way  to  keep  project  participants  interested.  Dispose  of 
research  methods  that  are  time-consuming  and  opt  for  ones 
that,  while  less  rigorous,  will  produce  reasonably  valid 
results  with  less  time. 

Lesson  9 

Willima  of  Ochen  of  the  middle  ages  gave  us  this  lesson: 
"keep  it  as  simple  as  possible,  but  no  simpler."  SumOne  for 
Kids  aimed  to  create  a "simple"  product&endash;one  that 
was  useful  and  easy  to  use.  The  final  product  is  far  bigger 
than  originally  thought  necessary.  It  was  perhaps  inevitable 
because  this  first-of-its-kind  product  used  a consensus 
model  involving  over  3 1 agencies  and  other  stakeholders  in 
the  design  and  development.  The  first  "final"  product 
(which  includes  a comprehensive  database  on  describing 
the  children  and  families  served,  services  received,  and 
results  produced  in  five  major  functional  outcome  areas) 
has  been  met  with  enthusiasm  by  those  who  have  used  it  but 
with  concerns  by  those  who  see  it  as  too  much.  Our 
interaction  with  others  who  are  developing  outcome 
measurement  systems  indicates  that  the  move  to  complex  is 
quicker  than  the  move  to  elegant  simplicity.  Future 
developments  will  be  a balance  of  adding  more  to  the 
system  (such  as  "protective  factors"  and  eliminating 
complicated  protocols  and  unused  data. 

Discussion 
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The  project  proved  sufficiently  successful  to  be  spun-off 
into  a separate  corporation  called  the  Corporation  for 
Standards  and  Outcomes  (CS&O).  CS&O  is  now 
replicating  and  improving  upon  SumOne  for  Kids  in 
Maryland  through  the  Maryland  association  of  65 
child-serving  agencies  in  that  state  and  has  made 
participation  in  SumOne  for  Kids  mandatory  for  agency 
members.  Given  the  multi-agency,  multiple  systems 
represented  by  these  provider  agencies,  and  statewide 
nature  of  the  SumOne  for  Kids  outcome  measurement 
system,  staff  believe  that  the  results  will  ultimately  have  a 
powerful  impact  on  children's  services  in  Pennsylvania  and 
Maryland. 
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The  Building  Outcome  Accountability  in  Children's  Mental 
Health  project  was  an  effort  to  learn  more  about  how  each 
of  the  twenty-two  Center  for  Mental  Health  Services 
(CMHS)  grantees  were  conceptualizing  and  implementing  a 
local  process  of  outcome  evaluation.  This  study  was 
conducted  as  part  of  the  System  Accountability  Study,  a 
5-year  research  effort  of  the  Research  and  Training  Center 
for  Children's  Mental  Health  (RTC)  designed  to  investigate 
the  impact  that  utilizing  measurable  outcomes  has  on 
service  systems. 

The  Building  Outcome  Accountability  project  focused  on 
the  conceptualization  and  implementation  of  outcome-based 
information  systems  at  the  twenty-two  CMHS  sites.  These 
sites  received  grants  designed  to  promote  the  development 
of  systems  of  care  that  include  mental  health,  child  welfare, 
education,  juvenile  justice,  and  other  appropriate  agencies  to 
meet  the  multiple  and  changing  needs  of  children  and 
adolescents  and  their  families.  A related  activity,  the  core 
evaluation  of  the  CMHS  initiative,  conducted  by  Macro 
International,  Inc.  and  its  partner,  the  University  of  South 
Florida,  is  focused  on  the  child,  the  system,  and  the 
interaction  between  the  two.  The  Building  Outcome 
Accountability  project  focused  on  how  outcome  results 
were  being  used  in  CMHS  systems  of  care  in  a effort  to 
understand  how  using  measurable  outcome  data  affects 
service  planning  and  delivery. 
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Methodology 

The  data  for  the  Building  Outcome  Accountability  project 
were  collected  through  a series  of  structured  key  informant 
telephone  interviews  with  employees  at  each  of  the  CMHS 
sites.  Two  interview  protocols  were  developed  for  this 
research  project:  1)  Survey  A,  designed  for  sites  with 
established  outcome  information  systems;  and  2)  Survey  B, 
designed  for  sites  which  had  not  yet  established  outcome 
information  system.  Each  site  determined  who  would 
participate  in  the  interview  and  whether  Survey  A or  Survey 
B would  be  completed. 

Both  survey  instruments  asked  participants  to  comment  on 
the  array  of  services  offered  by  their  system  of  care,  how 
and  when  the  system  of  care  was  established,  and 
demographic  information  about  the  population  served.  Both 
surveys  also  asked  participants  to  comment  on  the 
involvement  of  stakeholders  in  the  process  of  selecting 
outcomes  to  be  tracked  and  the  impact  of  outcome  tracking 
on  service  planning  and  delivery.  In  addition,  systems  which 
identified  themselves  as  having  established  outcome 
tracking  systems  were  asked  to  comment  on  what  kinds  of 
outcomes  were  tracked,  how  often  they  are  tracked,  how 
these  outcomes  are  measured,  and  the  level  of  interagency 
collaboration  required  to  monitor  outcomes.  They  were  also 
asked  to  discuss  how  outcome  information  is  being 
reported,  who  receives  these  reports,  and  how  it  is  being 
used.  Finally,  respondents  that  completed  the  survey 
questions  for  sites  with  an  established  outcome  information 
system  were  asked  to  comment  on  the  continued  use  of 
outcome  data  when  the  national  evaluation  is  completed. 

The  interviews  for  this  project  were  conducted  during  a 
five-month  period  beginning  October  1995  and  ending  in 
February  1996.  Nineteen  out  of  the  twenty-two  grantee  sites 
(86%)  participated  in  the  interview  process.  All  participants 
were  advised  that  the  results  of  this  study  would  be  reported 
in  aggregated  form  and  that  the  identities  of  individual 
systems  of  care  and  respondents  would  remain  anonymous. 
Ten  of  the  respondents  (53%)  represented  sites  with  an 
established  outcome  information  system,  while  nine 
participating  sites  (47%)  did  not  have  such  a system 
established. 

Results 

The  data  was  analyzed  across  three  domains:  1)  stakeholder 
involvement  in  selecting  outcomes;  2)  the  impact  of 
outcomes  on  service  planning  and  delivery;  and  3)  using 
outcomes  when  the  national  evaluation  is  completed.  The 
results  are  discussed  below. 

Stakeholder  Involvement 

Participants  were  asked  to  rate  the  importance  of 
stakeholder  involvement  on  a Likert-type  scale  from  1-  not 
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stakeholder  involvement  on  a Likert-type  scale  from  1 - not 
important  at  all  to  5-  crucial.  All  but  one  of  the 
participating  sites  considered  stakeholder  involvement  in  the 
process  of  selecting  outcomes  to  be  veiy  important  or 
higher.  The  single  site  not  responding  in  this  range  reported 
that  they  chose  not  to  respond,  believing  it  to  be  irrelevant 
to  their  situation  because  they  had  uniformly  adopted  the 
outcome  measures  currently  being  used  in  the  national 
evaluation.  While  nineteen  sites  considered  stakeholder 
involvement  important  to  the  process  of  selecting  outcomes, 
5 sites  (one  site  which  considered  stakeholder  involvement 
very  important  and  four  which  considered  it  crucial ) 
reported  having  no  vehicle  currently  developed  to  facilitate 
this  involvement.  One  respondent  commented  that  there  is 
no  systematic  way  of  involving  stakeholders  at  this  time, 
although  they  anticipated  doing  so  in  the  future.  Another 
commented  that  there  is  a "growing  discussion  of  how 
stakeholders  can  be  involved."  These  results  imply  that 
while  almost  all  of  the  respondents  thought  stakeholder 
involvement  was  important,  some  sites  (26%)  had  no 
mechanism  for  achieving  this  goal.  This  indicates  that 
agreement  with  the  principle  does  not  always  equate  to 
implementation. 

Impact  on  Service  Planning  and  Delivery 

Nineteen  of  the  twenty  participants  in  this  study  chose  to 
answer  the  series  of  questions  relating  to  the  impact  of 
outcome  tracking  on  service  planning  and  delivery.  One  site 
declined  because  they  believed  there  was  not  yet  enough 
outcome  data  available  yet  to  comment.  Respondents  were 
asked  to  rank  the  impact  of  outcome 
tracking&endash;anticipated  or  actual&endash;on  service 
planning  and  service  delivery  on  a Likert-type  scale  of  1-  «o 
impact  to  5 - tremendous  impact.  Ten  participants  (53%) 
responded  that  the  impact  on  services  planning  was 
tremendous,  eight  responded  the  impact  was  4 - 
considerable,  and  one  responded  outcome  tracking  had  3 - 
some  impact.  The  ten  sites  which  rated  the  impact  of 
outcomes  on  service  planning  as  tremendous  also  believed 
the  impact  of  outcome  tracking  on  service  delivery  to  be 
tremendous.  Of  the  remaining  nine  sites,  six  rated  the 
impact  of  outcomes  on  service  delivery  as  considerable,  and 
three  believed  outcomes  would  have  some  impact  on  service 
planning. 

A common  concern  expressed  by  respondents  was  for  the 
burden  and  consequent  stress  that  the  process  of  establishing 
a system  of  outcome  information  places  on  management  and 
staff.  One  respondent  commented,  "Putting  an  outcome 
system  in  place  can  be  painful  because  of  the  procedural 
changes,  increases  in  paperwork,  and  increased  need  for 
management  and  staff  training."  Other  respondents  echoed 
this  statement.  Concern  was  expressed  that,  "outcome 
tracking  adds  paperwork  and  takes  time  away  from  actual 
service  provision."  Another  commented  that  "tracking 
outcomes  requires  ongoing  training  and  technical  assistance 
on  how  to  and  why  outcomes  are  collected.  People  are 
intimidated  by  data  and  forms.  You  must  keep  reinforcing 
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intimidated  by  data  and  forms.  You  must  keep  reinforcing 
the  process." 

The  experience  of  staff  at  CMHS  sites  which  have 
established  outcome  information  systems  indicates  that  the 
stress  and  burden  of  implementation  may  be  a short-term 
concern.  One  respondent  discussed  the  short  term  impact: 
"Service  providers  are  heavily  impacted  by  the  demands  of 
the  outcome  process.  It  is  difficult  to  implement  in  the 
short-run  because  the  most  immediate  impact  is  that 
clinicians  are  overwhelmed  by  the  process."  According  to 
some  respondents,  however,  once  outcome  information 
becomes  available,  stress-levels  are  reduced. 

Several  respondents  expressed  concern  for  fears  associated 
with  how  outcome  information  is  used.  One  respondent 
stated,  "there  exists  a real  fear  on  the  part  of  clinicians  that 
outcomes  will  be  used  to  demonstrate  they  are  not  doing  a 
good  job  &endash;  what  if  we're  not  doing  well?"  There 
was  consensus  that  training  was  an  important  way  to 
promote  understanding  of  how  outcomes  can  be  used  in  a 
positive  way  to  improve  service  delivery. 

Continuation  of  Outcome  Monitoring 

Respondents  representing  the  ten  sites  with  established 
outcome  information  systems  were  asked  a series  of 
questions  about  whether,  and  under  what  circumstances, 
they  might  continue  tracking  outcome  information  after 
completion  of  the  national  evaluation.  All  ten  responded 
that  they  anticipate  continuing  the  process  of  outcome 
monitoring,  although  the  strength  of  the  replies  ranged  from 
"Yes,  hopefully,  it  depends  on  available  funding"  to  "Yes, 
definitely,  outcomes  very  similar  to  the  national  evaluation 
are  included  in  our  state  plan."  When  asked  to  identify 
factors  that  would  influence  the  decision  to  continue 
outcome  tracking,  it  is  noteworthy  that  eight  of  the  ten  sites 
mentioned  available  funding  as  the  central  deciding  factor. 

Conclusions 

It  was  recognized  early  in  the  development  of  the  interview 
instruments  used  in  this  project  that  there  would  be  variation 
among  the  CMHS  grantees  in  the  degree  to  which  their 
systems  of  care  had  been  conceptualized  and  implemented, 
as  well  as  the  degree  to  which  their  information 
accountability  systems  were  developed.  The  amount  of 
overlap  between  the  level  of  development  of  the  service 
system  and  the  outcome  information  system  seemed  to  have 
an  impact  on  how  outcome  information  is  used  by  the 
system  of  care  fsee  Figure  U.  This  can  be  seen  in  how 
information  about  clinical  outcome  measures  is  used. 
Descriptions  of  three  example  sites  follow  to  illustrate 
varying  degrees  of  overlap  between  the  level  of 
development  of  the  service  systems  and  the  accountability 
systems  in  this  study. 

A respondent  whose  system  of  care  began  developing 
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A respondent  whose  system  of  care  began  developing 
within  the  past  few  years  reports  that  the  site  is  also  just 
beginning  to  collect  outcome  information.  This  site 
respondent  indicates  that  the  system  does  not  have  a vehicle 
for  involving  stakeholders  in  decisions  regarding  outcomes 
at  this  time.  Interagency  collaboration  at  the  site  occurs  only 
at  the  direct-service  level  where  interagency  teams  serve 
individual  children  and  their  families.  Although  this 
respondent  noted  that  the  "CAP AS  [Child  and  Adolescent 
Functional  Assessment  Scale]  is  an  excellent  measure,"  staff 
are  still  struggling  with  trying  to  determine  what  outcomes 
they  need  to  track  and  how  to  use  the  results.  This  system 
seems  to  represent  a situation  in  which  the  service  system 
and  the  outcome  accountability  system  are  both  developing 
in  parallel  and  exist  independently  of  one  another  at  this 
point  in  time. 

Another  grantee  in  the  early  stages  of  system  development 
describes  their  system  as  having  "lots  of  pieces  for  the  past 
eight  or  so  years,  but  we  only  recently  began  building  an 
integrated  system."  This  respondent  describes  the  site  as 
relying  heavily  on  the  input  of  a broad  range  of  stakeholders 
as  it  expands  its  array  of  services.  Also,  interagency 
relationships  are  growing  in  strength  and  he/she  believes 
that  "to  take  a systems  point  of  view,  we  need  to  know  what 
our  connection  with  other  systems  is."  This  respondent 
reports  that  the  site  is  already  using  clinical  data  to  make 
decisions  at  the  treatment  level,  but  does  not  yet  have  a way 
of  using  this  information  at  a program  or  system  level.  This 
site  represents  a system  at  the  early  stages  of  both  system 
and  information  development.  Some  degree  of  overlap  in 
the  development  of  the  services  offered  and  the  information 
system  seems  to  be  indicated  by  the  use  of  clinical  outcome 
data  to  inform  treatment  at  the  child  and  family  level. 

In  contrast  to  the  previous  examples,  one  of  the  more 
developed  systems  of  care  sites  has  already  focused  on 
potential  uses  for  outcome  results  or  information  at  the 
system  level.  The  respondent  for  this  site  reported  that  its 
system  of  care  had  been  developed  for  more  than  ten  years, 
and  it  had  been  measuring  and  reporting  outcome 
information  for  several  years.  The  initial  outcomes  tracked 
through  the  system  of  care  were  more  global;  for  example, 
emphasis  was  placed  on  system  level  measures  such  as 
out-of  home  placements,  school  attendance  and 
achievement,  and  Juvenile  Justice  recidivism  rather  than 
clinical  measures.  Discussions  at  the  site  concerning 
different  aspects  of  the  information  system  focused  on 
issues  of  sharing  information  across  the  system,  such  as 
creating  electronically  connected  information  systems  to 
lower  technical  boundaries  so  that  outcome  data  could  be 
accessed.  This  site's  focus  on  clinical  outcome  measures 
went  beyond  their  use  at  a child  level/direct  service  level  to 
recognition  of  how  clinical  status  information  might  be  used 
to  learn  more  about  how  effectively  the  system  is 
functioning.  This  site  seems  to  represent  a system  of  care 
which  is  well  developed  with  an  information  system  which 
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is  well  established.  In  addition,  there  seems  to  be  a high 
degree  of  overlap  between  the  system  of  services  and  the 
information  system.  This  takes  place  through  its  efforts  to 
create  a seamless  source  of  interagency  information  and  its 
interest  in  using  clinical  outcome  information  to  evaluate 
system  effectiveness  in  addition  to  informing  treatment 
decisions. 

In  conclusion,  the  CMHS  sites  represent  varying  degrees  of 
both  systems  development  outcome  accountability 
development.  For  outcomes  to  have  a maximum  positive 
impact  on  service  planning  and  delivery,  it  may  be 
necessary  for  there  to  be  a high  degree  of  overlap  between 
the  service  system  and  the  information  system.  The  data 
collected  in  this  study  illustrate  a range  of  overlap  from  a 
service  system  which  seems  to  exist  independently  of  the 
information  system  being  developed  to  support  it,  to  one 
which  seems  to  have  a high  degree  of  overlap,  as  evidenced 
by  the  multiple  uses  of  clinical  outcome  measures  for 
informing  service  planning  and  delivery. 
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Introduction 


Mental  health  decision  makers  need  comprehensive, 
uniform  data  in  order  to  conduct  analysis  and  planning  on 
the  national  and  state  levels.  A major  initiative  to  promote 
uniform  mental  health  data  began  in  1989  when  the 
federally  funded  Mental  Health  Statistics  Improvement 
Project  (MHSIP)  defined  a set  of  standard  data  elements 
recommended  for  mental  health  information  systems 
(NIMH,  1989).  Following  the  publication  of  the  MHSIP 
standards,  the  Task  Force  on  Enhancing  MHSIP  to  Meet  the 
Needs  of  Children  and  Youth  in  1992  recommended 
additional  MHSIP  data  elements  specific  to  children's 
mental  health. 

States  are  still  struggling  to  fully  implement  MHSIP 
standards  in  their  mental  health  information  systems.  One  of 
the  obstacles  seems  to  be  a perceived  conflict  or 
competition  between  agency  information  systems'  priorities 
and  the  requirements  of  the  MHSIP  standards. 

Agency  managers'  top  priority  is  to  have  a system  that 
provides  data  for  management  of  clients  and  programs.  In 
addition,  they  periodically  need  specific  data  to  respond  to 
evaluation  inquiries.  If  evaluation  data  are  not  available, 
then  special  data  collections  have  to  be  carried  out  to  meet 
each  data  request.  If  collecting  the  information  to  meet 
either  of  these  goals  appears  to  conflict  with  implementing 
the  MHSIP  standards,  agency  needs  often  take  priority  over 
the  MHSIP  standards. 

An  additional  obstacle  can  arise  when  mental  health 
agencies  wish  to  share  data  with  external  multi-agency 
children's  data  bases.  This  requires  the  mental  health  agency 
to  have  the  appropriate  data  in  a format  acceptable  to  other 
agencies.  The  value  of  sharing  data  with  larger  data  bases 
may  be  perceived  to  supersede  the  importance  of  meeting 
MHSIP  standards. 
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If  children's  mental  health  administrators  perceive  the 
MHSIP  standards  to  be  an  additional  burden  that  uses 
resources  they  would  rather  be  using  to  meet  important  data 
needs,  then  MHSIP  will  not  be  implemented.  We  all 
recognize  that  no  information  system  will  be  able  to  meet 
every  need,  but  the  ideal  is  a system  that  provides  as  much 
of  the  necessary  data  as  possible  so  that  special  data 
collections  are  rare. 

The  purpose  of  this  study  was  to  examine  the  extent  to 
which  children's  mental  health  agencies  can  meet  the 
information  needs  they  consider  the  most  important  and  still 
support  the  MHSIP  standards.  Since  the  focus  of  the  study 
was  on  agency  information  decisions,  the  data  sources  and 
study  questions  were  selected  to  reflect  real  life  situations. 

Method 

Three  children's  data  sets  were  selected  for  this  analysis  to 
represent  three  different  agency  information  priorities: 

• An  information  system  for  client  and  program 
management  to  represent  the  data  needs  for  agency 
management. 

• The  data  set  for  a national  children's  mental  health 
program  evaluation  to  identify  typical  evaluation  data 
needs. 

• A social  service  agency  data  set  for  children  involved 
in  foster  care  and  adoption  to  illustrate  the  data  needs 
of  a different  type  of  child-serving  agency. 

The  analysis  used  real  life  scenarios,  with  the  questions 
framed  from  the  point  of  view  of  an  agency  administrator 
making  decisions  about  information  systems.  Each  question 
asked  if  an  agency  had  an  information  system  designed  to 
meet  one  purpose,  what  alterations  would  be  required  to 
expand  the  system  to  meet  an  additional  purpose. 

• If  a children's  mental  health  agency  has  an 
information  system  that  meets  their  management 
needs,  what  changes  would  have  to  be  made  for  the 
system  to  also  meet  MHSIP  standards? 

• If  an  mental  health  agency  has  an  information  system 
that  meets  MHSIP  standards,  what  changes  would  be 
needed  for  it  to  also  provide  evaluation  data? 

• If  a social  service  agency  is  developing  a foster  care 
and  adoption  information  system,  what  changes 
would  a mental  health  agency  have  to  make  for  the 
system  also  to  meet  MHSIP  standards? 

In  order  to  compare  the  data  elements  across  systems,  the 
client  and  event  data  elements  in  the  MHSIP  standards  and 
the  selected  data  sets  were  subdivided  into  the  following 
domains:  (a)  Program  Information,  (b)  Client 
Demographics,  (c)  Family  Structure/Placement,  (d) 

Referral,  (e)  Health/Mental  Health  Status,  (f)  Plan  of  Care, 
and  (g)  Outcomes.  Program  Information  included  record 
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and  (g)  Outcomes.  Program  Information  included  record 
number,  state  of  residence,  and  dates  of  client's  enrollment 
or  discharge  from  the  program.  Client  Demographics 
contained  basic  client  demographic  elements  such  as  name, 
social  security  number,  sex,  race,  and  insurance.  Family 
Structure/Placement  included  information  on  current  and 
historical  family  structures  &endash;  including  birth 
parents,  foster  and  adoptive  parents,  and  other  caretakers 
&endash;  as  well  as  placement  information.  Referral 
included  information  about  referral  source,  intake,  previous 
involvement  of  other  agencies  and  presenting  problem. 
Health/Mental  Health  Status  captured  assessment  tools  and 
other  indicators  of  health  and  functioning  including  school 
placement.  Plan  of  Care  addressed  services  information  and 
multi-agency  involvement,  and  Outcomes  included 
indicators  of  client  status  over  time  and  at  termination  from 
the  program. 

Results 

This  study  found  that  the  MHSIP  standards  were  very 
compatible  with  mental  health  agency  needs  for  program 
and  client  management  data,  as  well  as  with  needs  for 
evaluation  data.  In  addition,  when  agencies  wish  to  merge 
their  data  into  larger  children's  data  bases  outside  the 
agency,  with  the  exception  of  data  elements  specific  to 
mental  health  operations,  a high  degree  of  compatibility 
between  these  data  bases  and  the  MHSIP  standards  can  be 
expected.  More  specifically,  the  analysis  suggests: 

• Agencies  with  typical  management  information 
systems  should  be  able  to  meet  MHSIP  standards 
with  minimal  effort. 

• MHSIP-based  agency  information  systems  show 
substantial  agreement  with  a typical  evaluation  data 
set  except  for  some  missing  data  elements  in  the 
domains  of  Family  Structure/Placement  and 
Outcomes 

• In  spite  of  the  differences  in  their  missions  and 
purposes,  social  service  agency  data  bases  contain  a 
substantial  number  of  the  same  data  elements  as 
children's  mental  health  data  bases,  with  only  the  data 
elements  specific  to  the  mental  health  activity 
missing. 

Discussion 

Children's  mental  health  agencies  attempt  to  be  part  of 
several  worlds.  Within  the  comprehensive  mental  health 
system,  they  need  to  cooperate  with  the  data  collection  and 
reporting  strategies  of  that  system.  As  part  of  a total  system 
of  care  for  children,  they  need  to  cooperate  in  data  sharing 
with  the  other  agencies  and  partners  who  serve  children  and 
families.  While  trying  to  accommodate  the  needs  of  both  of 
these  complex  systems,  they  need  to  manage  the  programs 
and  clients  for  which  they  are  responsible  and  regularly 
respond  to  internal  reporting  and  evaluation  needs.. 
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Achieving  an  information  system  that  specifically  meets 
each  of  these  demands,  and  yet  is  flexible  enough  to 
accommodate  all  of  them,  is  challenging.  This  analysis 
found  that  there  is  a high  level  of  overlap  in  data  elements 
across  systems  representing  each  of  the  requirements. 
Therefore,  designing  a system  which  will  meet  several  of 
the  needs,  or  alternatively  expanding  an  existing  system  to 
accommodate  new  purposes,  need  not  be  an  insurmountable 
task. 
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Failure  to  ground  the  wraparound  model  in  more  than 
value-based  philosophic  principles  is  leading  to  its 
adaptation  as  another  form  of  case  management  by 
professionals  whose  training  and  experience  has  been  in 
expert  models  of  deficit  remediation.  These  models  limit  the 
decision-making  voice  of  families  while  ignoring  or  failing 
to  utilize  their  perspectives  and  strengths.  The  following 
summaries  describe  thirty  months  of  participatory  program 
evaluation  and  simultaneous  program  development  between 
the  University  of  South  Florida's  Child  and  Family  Policy 
Program  and  a mental  health-supported,  elementary 
school-based  program  in  Tampa,  FL,  and  begin  to  address 
the  critical  question  of  fidelity  of  intervention  and  its 
relationship  to  outcomes  for  children  and  their  families. 

The  Joint  Venture  Family  School  Support  Team  (FASST) 
implemented  a "family-centered,  strengths-based"  approach 
which  they  called  wraparound.  However,  key  informant 
interviews  conducted  in  November,  1993  with  eight  policy 
makers  from  the  Hillsborough  County  Public  Schools  and 
the  Children's  Board  of  Hillsborough  County  indicated  that 
many  lessons  learned  from  the  Ventura  Project  (Jordan  & 
Hernandez,  1990)  had  not  been  applied.  There  appeared  to 
be  no  agreement  on  target  population,  no  consensus  on 
desired  system  change,  and  no  common  understanding  of  . 
the  wraparound  model  implemented  through  the  FASST 
program.  These  summaries  describe  how  simultaneous, 
participatory  processes  of  evaluation  and  program 
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participatory  processes  of  evaluation  and  program 
development  defined  the  wraparound  model,  measured 
outcomes,  and  identified  elements  of  program  policy, 
management,  and  practice  that  were  undermining  the 
integrity  of  the  intervention. 
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Deciphering  the  Tower  of  Babel:  Preliminary 
Steps  Toward  Establishing  a Theory  Base  for 
Wraparound  Fidelity 

Rosalyn  Malysiak,  Ph.D. 

Introduction 

Multiple  theories,  methodologies,  and  confused  terminology 
rooted  in  different  paradigms  of  thought  have  contributed  to 
an  inadequate  service  structure  in  which  professionals 
utilize  medically  driven  models  as  the  perceptual  prism  to 
define  families  of  children  with  serious  emotional 
disturbance  as  dysfunctional.  European  in  origin,  a North 
American  alternative  to  this  expert  model  is  emerging  which 
has  been  loosely  termed  "wraparound  process,"  and  which 
has  coalesced  around  a broadly  stated,  strengths-based  and 
family-centered  ecological  approach,  emphasizing 
individualized  service  and  treatment  in  the  most  appropriate 
and  least  restrictive  setting  (Boyd,  1991;  Burchard  & 

Clarke,  1990;  Duchnowski  & Friedman,  1990;  VanDenBerg 
& Grealish,  1996). 

Applications  of  wraparound  are  now  emerging  from  an 
early  developmental  stage  in  which  it  has  been  defined 
through  value-based,  philosophic  principles  that  begin  to 
differentiate  it  from  the  professionally  driven  process 
characteristic  of  more  traditional  forms  of  family-centered 
practice.  However,  the  maturation  of  wraparound  is 
threatened  by  a developmental  paradox.  Those  who  have 
been  trained  in  the  old  theories  of  assessment  and 
remediation  of  deficit,  and  whose  careers  have  been  shaped 
by  the  professionally  driven  process  of  traditional  service 
models,  must  play  transformative  roles  in  the  emergence  of 
this  promising  alternative.  This  potential  conflict  is 
exacerbated  by,  and  echoes  through,  a "Tower  of  Babel"  of 
terminology  used  to  describe  seemingly  similar  approaches 
to  working  with  children  and  their  families.  Amidst  this 
confusion,  the  value-based  principles  which  have  guided 
wraparound  have  all  too  often  been  misinterpreted  and 
misapplied  as  emergent  case  management  methodology. 

This  essential  change  from  categorical  to  integrated  and 
individualized  services,  and  from  deficit  assessment  by 
professionals  to  a process  of  ecological  strengths 
enhancement  which  engages  families  as  decision-making 


150 


enhancement  which  engages  families  as  decision-making 
participants,  has  occurred  without  articulation  of  its  implicit 
roots  in  constructivist  and  critical  thought,  nor  of  its  slightly 
more  explicit  basis  in  ecological  systems  theory.  This 
summary  presents  initial  data  from  a case  study  design  (Yin, 
1 989;  1 994)  which  evaluated,  as  a basis  for  wraparound 
fidelity,  a single  construct  and  operative  focus:  families 
acting  as  decision  making  participants  in  a process  of 
ecological  strengths  enhancement. 

This  descriptive,  exploratory  study  generalized  to  theory, 
not  to  a population.  It  contended  that  when  consciously 
applied  in  tandem,  elements  of  this  construct  anchor  the 
wraparound  process  in  its  implicit  basis  of  constructivist 
and  critical  thought  and  ecological  systems  theory.  Without 
their  application  from  this  basis,  the  wraparound  process 
reverts  toward  professionally  driven  and  deficit-focused 
efforts  typical  of  more  traditional  forms  of  family-centered 
practice.  Cases  in  this  study  were  defined  as  all  participants 
in  the  development  and  implementation  of  family  support 
plans  in  the  FASST  program.  Seven  cases,  opened  in 
October  and  November,  1995,  formed  the  basis  for  the 
study  which  focused  through  experiences  of  participants  in 
the  purposively  sampled  cases. 

A critical  review  of  the  literature  focused  on  the  social  and 
paradigmatic  emergence  and  transformation  of  family 
systems  theory  and  the  related  development  and 
transformation  of  family-centered  practice  through  a wide 
range  of  disciplines.  This  review  differentiated  and 
operationalized  five  levels  of  family-centered  practice  as 
focused  through  the  complimentarity  of  family  and 
professional  roles  and  their  use  of  strengths  in  support 
planning  and  implementation  (see  Table  11.  The  first  three 
levels  reflected  more  traditional  forms  of  family-centered 
practice  within  expert  models  which  focused  upon  deficit 
remediation.  Levels  four  and  five  described  when  families 
acted  as  decision  making  participants  in  a process  of 
ecological  strengths  enhancement,  the  posited  basis  for 
wraparound  fidelity  as  a collaborative  model  of 
family-centered  practice. 

Method 

Two  opposing  sets  of  a priori  propositions  were  derived 
from  the  review  of  the  literature,  and  from  these,  operational 
definitions  of  family-centered  practice  were  applied  in  this 
study.  These  propositions  described  and  assessed  the 
presence  of  elements  of  these  two  models  in  the  seven  cases 
through  three  methods  of  data  collection  at  different  points 
in  the  development  and  implementation  of  the  wraparound 
plans.  The  patterns  which  emerged  at  the  conclusion  of  the 
study  in  May,  1996  addressed  the  research  question:  "When 
families  act  as  decision  making  participants  in  a process  of 
ecological  strengths  enhancement,  how  do  applications  of 
wraparound,  a collaborative  model,  differ  from 
family-centered  practice  within  the  expert  model?"  The 
propositions  tested  for  wraparound  fidelity  as  a 
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propositions  tested  for  wraparound  fidelity  as  a 
collaborative  model  of  family-centered  practice  were: 

• Families  will  act  as  participants  of  a community  team 
in  which  assessment  and  implementation  decisions 
are  reached  by  consensus.  VvTien  consensus  cannot  be 
reached,  the  team  will  value  and  abide  by  the  decision 
of  the  family. 

• Community  participants  will  share  their  expertise  and 
perspective  in  a way  which  provides  information  as 
well  as  access  to  service  while  respecting  the  families' 
perspectives. 

• Families  will  have  sufficient  information  as  well  as 
access  to  services  to  voice  their  perspectives, 
strengths,  and  needs. 

• Decisions  made  by  this  team  will  recognize  and 
combine  the  strengths  of  the  family  with  the  strengths 
in  the  community,  including  the  more  traditional 
service  structure. 

• The  combined  ecological  and  family  strengths  will  be 
the  basis  for  individualized  activities  which  target 
specific  needs. 

Propositions  tested  for  family-centered  practice  within  an 
expert  model  were: 

• Families  will  provide  information  to  professionals 
who  assess  problems  or  needs  and  who  then  attempt 
to  remedy  them  by  matching  each  to  an  existing 
service. 

• Community  participants  will  present  their  perspective 
and  expertise  in  a manner  which  limits  the  role  of  the 
family  as  an  equal  decision  making  participant  and 
which  does  not  respect  the  family's  perspective. 

• Services  delivered  to  the  family  will  not  reflect  the 
family's  perspective  of  what  might  best  meet  their 
needs. 

• If  strengths  are  identified,  they  will  primarily  be 
within  the  family,  and  they  will  not  be  actively  and 
overtly  utilized  to  meet  identified  needs. 

• Activities  in  the  family  support  plans  will  not  overtly 
seek  to  combine  strengths  of  the  family  with  strengths 
in  the  community. 

The  study  employed  a multi-method,  multi-source  approach 
with  three  primary  sources  or  waves  of  data:  (a)  systematic 
observation  of  community  team  meetings  in  which  issues 
and  perspectives  were  explored  with  referred  families,  and 
in  which  the  family  support  plan  was  developed  (October  «& 
November,  1995);  (b)  semi-structured  interviews  with 
family  support  plan  participants  involved  in  the  seven  cases 
under  study  (January  through  March,  1996;  n = 44);  and  (c) 
systematic  review  of  the  formal  FASST  case  files.  Based  on 
the  initial  analysis  of  the  first  wave  of  data,  three  cases  were 
assigned  to  a collaborative  model  cell.  This  meant  that  at 
assessment  and  planning,  the  process  between  family 
members  and  other  participants  appeared  to  fit  operational 
definitions  4 or  5 of  family-centered  practice,  the  posited 
basis  for  wraparound  fidelity  (see  Table  IT  Four  cases  were 
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basis  for  wraparound  fidelity  (see  Table  IV  Four  cases  were 
assigned  to  an  expert  model  cell  because  coding  indicated 
that  assessment  and  planning  appeared  within  operational 
definitions  1,  2,  or  3 of  family-centered  practice  (see  Table 
1). 

On  a case  by  case  basis,  wave  2 semi-structured  interviews 
would  be  analyzed  and  coded  for  evidence  of  all 
propositions,  while  the  systematic  review  of  FASST  case 
files  in  Wave  3 would  be  utilized  to  corroborate,  disconfirm, 
or  enrich  data  which  emerged  from  the  first  two  sources. 
Through  this  within-case  analysis,  movement  from  one  cell 
at  assessment  and  planning  toward  the  other  cell  during 
implementation  could  be  documented.  This  multi-source, 
multi-method  design  would  be  utilized  to  develop 
converging  lines  of  inquiry  through  identification  of  pattern 
convergence  or  divergence  through  triangulation  or 
replication  in  all  waves  of  data.  Patterns  between  cases  in 
the  same  cell  would  be  examined  in  similar  manner  in  an 
iterative  process  to  develop  converging  lines  of  inquiry  to 
delimit  multiple  explanations  or  to  uncover  a few 
explanations  which  would  hold  under  predictable  situations. 

Study  Implications 

The  final  level  of  analysis  of  between  cell  patterns  would  be 
compared  and  contrasted  to  answer  the  research  question: 
"When  families  act  as  decision  making  participants  in  a 
process  of  ecological  strengths  enhancement,  how  do 
applications  of  wraparound,  a collaborative  model,  differ 
from  family-centered  practice  within  the  expert  model?" 

The  answers  to  this  question  will  begin  to  establish  a theory 
and  paradigm  basis  which  may  better  ensure  fidelity  than 
the  easily  misinterpreted  value-based  principles  which 
currently  define  wraparound.  This  will  provide  a 
preliminary  and  essential  step  toward  promoting  the 
integrity  of  wraparound  applications.  As  such,  it  may 
provide  greater  clarity  in  conceptualizing  frameworks  for 
more  successful  collaboration  between  families,  schools, 
and  communities  as  well  as  provide  a foundation  and  means 
for  process  evaluations  and  program  development  of 
wraparound  initiatives.  By  developing  a foundation  and 
means  for  ensuring  wraparound  fidelity,  outcome 
evaluations,  cost-benefit  analysis,  and  surveys  of  participant 
satisfaction  could  be  more  meaningfully  compared  with 
more  traditional  forms  of  family-centered  practice  and 
service  delivery  to  better  guide  policy  and  funding 
decisions.  Final  results  from  this  study  will  be  forthcoming. 
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Establishing  Wraparound  Fidelity  Through 
Participatory  Evaluation 

Rosalyn  Malysiak,  Ph.D.,  A1  Duchnowski,  Ph.D.,  Marcia 
Black,  M.S.W.,  & Michael  Greeson 

Evaluation  of  treatment  fidelity  in  the  implementation  of  a 
wraparound  approach  has  presented  a challenge  to  the  field 
of  children's  mental  health  services.  This  challenge  is  due, 
in  large  part,  to  the  lack  of  an  articulated  theory  that  can 
offer  constructs  to  better  anchor  practice  within  this 
promising  model.  Without  demonstrated  fidelity,  program 
evaluations  produce  confused  and  sometimes  disheartening 
results. 

This  summary,  along  with  two  others,  addresses  the 
evaluation  of  the  Joint  Venture  Family  School  Support 
Team  (FAS ST),  an  emerging  school-based  program  in 
Tampa,  FL  that  employed  a wraparound  approach.  FASST's 
target  population  was  children  who  had  received  or  were  at 
risk  of  receiving  a diagnosis  for  emotional  or  behavioral 
disturbance.  The  program  was  implemented  by  two  mental 
health  centers  who  shared  responsibility  in  the  program  and 
delivered  service  through  six  elementary  schools  in  a 
predominately  low  income  area  of  Tampa.  It  was  guided  by 
a manager,  and  each  school  had  a team  composed  of  a 
family  support  coordinator  and  a paraprofessional  family 
advocate  who  had  a child  attending  that  school.  These  two 
staff  members  met  with  referred  families  to  develop  initial 
support  plans  which  were  brought  to  a standing  community 
team  that  met  on  a monthly  basis  for  refinement  and  review 
of  the  plans. 

The  FASST  program  began  operations  in  the  fall  of  1993 
with  a new  and  relatively  inexperienced  staff  who  received 
traditional  mental  health  in-service  training,  as  well  as  some 
exposure  to  interpretations  of  the  value-based  philosophy  of 
the  wraparound  approach.  Soon  thereafter,  doctoral  students 
and  the  director  of  the  University  of  South  Florida's  Child 
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and  the  director  of  the  University  of  South  Florida's  Child 
and  Family  Policy  Program  began  a process  of  participatory 
research  and  evaluation  which  guided  program 
development. 

Initial  participant  observations  of  the  community  team 
meetings  as  well  as  program  staff  meetings  had  shown  that 
well-intentioned,  enthusiastic  family  support  coordinators 
and  family  advocates  were  highly  confused  and  pulled  in 
different  directions  by  the  multiple  perspectives  and  actions 
of  school,  mental  health,  and  social  service  system 
participants  in  the  wraparound  plans.  Roles  in  the  program 
were  not  well  defined,  and  the  process  enacted  with  families 
appeared,  at  best,  like  team  case  management  within  an 
expert,  medically  driven  model. 

To  develop  a common  base  of  operations,  USF  doctoral 
students,  acting  as  consultants,  led  program  staff  through  a 
participatory  evaluation  of  their  roles  and  the  program 
which  asked  three  deceivingly  simple  questions:  (a)  What 
worked?  (Best  Practice);  (b)  What  didn't  work?  (Barriers  to 
Best  Practice);  and  (c)  \^at  did  they  need  to  make  the 
program  work  better?  The  answers  to  these  questions  guided 
subsequent  program  development.  What  was  working  was 
combined  with  staff  suggestions  for  improvements,  and 
approaches  to  address  the  barriers  to  best  practice.  This  was 
a consciously  constructed  isomorphic  process  of 
participatory  evaluation  and  planning  which  mimicked  the 
evocation  of  multiple  perspectives,  the  development  of 
consensus,  and  the  combining  of  strengths  to  meet  needs 
which  occurs  in  a wraparound  approach. 

By  the  end  of  the  fall  of  1994,  FAS  ST  staff  had  requested 
and  received  considerable  support  in  developing 
strengths-needs  based  wraparound  plans,  as  well  as  in  how 
to  engage  families  as  decision-making  participants. 
Combining  this  support  with  what  they  felt  were  the 
strengths  within  their  program,  FASST  staff  then  identified 
that  the  standing  community  teams  were  themselves 
undermining  the  integrity  of  a wraparound  approach.  Each 
community  team  had  representatives  from  the  school, 
mental  health,  and  social  service  agencies  who  brought  their 
professional  expertise  to  the  development  and  review  of  the 
family  support  plans  in  a manner  which  often  failed  to 
acknowledge  or  utilize  the  perspectives  and  strengths  of  the 
family,  and  limited  the  family  role  primarily  to  one  of 
informant  and  recipient  of  service.  Plans  which  emerged 
from  these  teams  typically  identified  needs  solely  in  the 
family  and  matched  them  to  existing  services. 

Facilitated  by  the  USF  consultants  in  the  winter  of  1995, 
FASST  staff  developed  a questionnaire  to  assess  the 
community  team  members'  understanding  of  the  roles  of 
participants  and  process  within  a wraparound  approach.  The 
questions  focused  on  a key  construct,  with  its  operative 
focus  which  was  in  the  process  of  development  by  one  of 
the  consultants  to  use  as  the  basis  for  conceptualizing  and 
testing  a theory  base  for  wraparound  fidelity:  Families 
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testing  a theory  base  for  wraparound  fidelity:  Families 
acting  as  decision-making  participants  in  a process  of 
ecological  strengths-needs  based  planning  and 
implementation. 

Survey  results  from  41  of  60  possible  respondents  were 
reviewed  by  the  entire  FAS  ST  staff  over  a period  of  one 
month  in  a participatory  process  facilitated  by  the  USF 
consultants.  This  participatory  approach  appeared  to 
enhance  staff  cohesion  and  understanding  of  the  key 
construct  which  differentiates  wraparound  as  an  emerging 
collaborative  model  of  family-centered  practice  from  more 
traditional  family-centered  practice  within  expert  models  of 
deficit  remediation.  However,  results  indicated  that 
professionals  on  these  community  teams  had  difficulty 
transcending  the  formal  training  they  had  received  in  deficit 
theories  and  the  expert  model  of  practice.  Though  these 
professionals  articulated  a role  for  families  as  "partners," 
during  the  planning  and  implementation,  they  primarily 
relegated  families  to  roles  of  informant  and  recipient  of 
service  recommended  by  the  team.  The  slippage  into  the 
expert  model  was  further  revealed  by  the  FASST  staffs 
rating  these  responses  on  a scale  measuring  strengths-based 
planning  that  placed  the  community  team  members  solidly 
within  a deficit  remediation  focus  typical  of  more  traditional 
forms  of  family-centered  practice. 

These  results  were  used  in  another  participatory  process 
facilitated  by  USF  consultants  in  which  FASST  staff 
developed  a program  brochure  which  described  the  intended 
roles  of  families  and  professionals  in  a process  of  ecological 
strengths-needs  based  assessment  and  implementation. 
Subsequent  outcome  and  process  evaluations  were 
conducted  and  are  reported  in  the  following  section. 

Preliminary  Outcomes  of  a School-based 
Wraparound  Program 

Norin  Dollard,  M.P.A.  & Robert  Slewzckowski,  M.A. 

The  Joint  Venture  Family  and  School  Support  Teams 
(FASST)  serves  children  identified  as  having  or  at-risk  of 
emotional  and  behavioral  disorders  and  their  families. 
Through  a team  planning  process,  a Family  Support  Plan  is 
developed  which  guides  the  Family  Coordinators  and 
Family  Advocates  in  assisting  families  to  attain  the  goals 
they  have  set  for  their  themselves  at  home,  in  school,  and  in 
the  community.  The  composition  of  teams,  which  includes 
parents,  school  personnel,  and  community  agency 
representatives,  is  intended  to  foster  consideration  of  the 
family's  strengths  across  the  ecology  of  the  family  system  in 
the  development  of  this  plan. 

The  funding  of  this  project  requires  that  descriptions  of  the 
children  and  families  served,  the  services  provided,  and 
assessed  outcomes  are  reported.  While  these  data  serve  an 
important  function  (i.e.,  reporting  to  the  funding 
organizations  and  developing  the  "habit"  of  systematic  data 
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organizations  and  developing  the  "habit"  of  systematic  data 
collection  for  program  staff),  data  collection  efforts  were  not 
primarily  focused  on  supporting  development  at  the 
program  or  clinical  level.  Consistent  with  the  participatory 
approach  taken  to  the  development  of  an  ecological 
strengths-enhancement  among  staffing  team  members, 
doctoral  students  worked  with  the  program  staff  to  clarify 
ways  in  which  data  could  support  program  development. 
Three  target  areas  were  identified  for  enhanced  assessment: 
(a)  description  of  children  enrolled,  (b)  analysis  of  service 
delivery  and  utilization  patterns,  and  (c)  outcomes  of 
children  enrolled  in  FASST. 

Description  of  Children  Served 

Part  of  the  efforts  of  the  FASST  program  staff  and 
university  staff  led  to  incorporation  of  standardized 
measures  into  the  intake  and  case  review  process  to  inform 
clinical  decision  making  as  well  as  to  serve  the  evaluation 
needs  of  the  program.  In  the  future,  data  for  all  children 
enrolled  in  the  program  at  enrollment,  six  months,  and 
discharge  will  be  available.  These  procedures  were  not  fully 
in  place  at  the  end  of  the  second  year  of  program  operation, 
hence  the  smaller  number  of  children  for  whom  data  is 
available. 

While  103  children  received  services  in  the  1995-6  fiscal 
year,  including  49  children  who  were  enrolled  in  this  time 
frame,  the  present  report  is  limited  to  discussion  of  those 
children  who  received  at  least  six  months  of  services  ^ = 
51).  The  majority  of  these  children  were  white  non-Hispanic 
(55%)  males  (78%).  At  the  time  of  the  initial  staffing,  the 
largest  group  of  children  served  was  in  the  first  grade 
(21.6%),  as  is  consistent  with  FASST's  preventative  focus. 
However,  a large  proportion  were  in  the  fourth  (17.6%), 
fifth  (19.6%),  and  sixth  grades  (1 1.8%).  Most  of  the 
children  (79%)  received  free  or  reduced  lunch  rates  that 
serve  as  an  indication  of  low  income  levels  for  these 
families.  Although  many  of  the  children  in  FASST  are 
identified  for  special  education  services,  only  20%  are  being 
served  in  settings  for  those  identified  as  having  emotional 
handicaps  or  serious  emotional  disturbance.  Furthermore, 
their  attendance  is  of  concern,  with  the  average  student 
missing  over  three  weeks  of  school  (16.2  days,  range  0-55 
days)  in  the  year  before  enrollment  in  FASST. 

The  children  enrolled  in  FASST  present  with  some  very 
challenging  behaviors  and  functional  impairments,  both  at 
home  and  in  school.  Parents  (64%)  most  commonly  identify 
externalizing  behaviors  as  being  an  impetus  for  referral. 
Similarly,  school  personnel  identify  non-compliant  behavior 
(62%)  as  the  most  frequent  reason  for  referral,  but  academic 
risk  factors,  such  as  being  below  grade  level  (42%)  are  also 
considered  in  referring  a child.  These  risk  factors  also  are 
documented  in  standardized  measures  completed  by  parents, 
teachers,  and  Family  Coordinators. 
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To  obtain  an  idea  of  problem  behaviors  exhibited  at  home 
and  in  school,  parents  were  asked  to  complete  the  Child 
Behavior  Checklist  (CBCL;  Achenbach,  1991a).  Teachers 
provided  similar  information  about  the  child's  school 
behavior  by  completing  the  Teacher  Report  Form  (TRF; 
Achenbach,  1991b).  The  results  showed  that  both  parents 
and  teachers  identified  approximately  two-thirds  as  falling 
in  the  clinical  range  on  the  Total  Problem  T-Score,  an 
indicator  of  global  functioning  (see  Figure  IT  Consistent 
with  the  behaviors  reported  in  the  referring  information,  the 
majority  of  children  displayed  externalizing  behaviors  both 
at  home  and  in  school.  Substantial  proportions,  however, 
also  scored  in  the  clinical  range  on  the  Internalizing 
T-scores. 

To  ensure  multiple  perspectives  of  a child's  level  of 
functioning.  Family  Coordinators  completed  the  Child  and 
Adolescent  Functional  Assessment  Scales  (CAFAS; 

Hodges,  1995).  The  Family  Coordinators  were  somewhat 
more  positive  about  the  child's  functional  status  than  parents 
and  teachers,  but  indicated  that  around  30%  of  the  children 
experienced  moderate  to  serious  impairment  in 
age-appropriate  role  performance  (48%),  behaviors  towards 
others  (29%),  and  in  moods/self-harm  (30%).  Family 
Coordinators  also  reported  that  many  of  the  caregivers 
generally  were  able  to  provide  materially  for  their  children; 
70%  of  the  families  evidenced  no  or  mild  impairment  in  this 
area.  Providing  a nurturing  home  environment  with 
adequate  family  and  social  supports  appeared  to  be  more 
challenging,  with  50%  of  parents  experiencing  moderate  to 
serious  impairment  in  this  area. 

Service  Delivery,  Utilization,  and  Costs 

There  were  two  agencies  that  contracted  to  provide  FAS  ST 
services.  Both  provided  the  same  types  of  services  and  both 
delivered  these  services  in  the  same  settings.  For  both 
agencies,  50%  of  the  services  were  delivered  in  the  home  or 
at  school.  The  breakdown  of  service  types  provided  includes 
the  following:  (a)  case  management  and  family  support 
services  of  the  Family  Coordinators  and  the  Family 
Advocates  (42%);  (b)  clinical  on-site  (i.e.,  in-home  therapy 
and  respite;  39%);  (c)  clinic-based  therapy  (18%);  and  (d) 
psychiatric  services  (1%). 

Related  to  understanding  the  types  of  services  and  where 
they  are  delivered,  is  to  understand  to  whom  and  in  what 
proportions  they  are  provided.  To  this  end,  an  analysis  of 
patterns  of  utilization  also  was  undertaken.  Program-wide,  it 
was  discovered  that  66%  of  case  management,  clinical,  and 
supportive  staff  hours  were  devoted  to  26%  of  children  and 
families.  This  pattern  was  remarkably  consistent  across  the 
six  schools.  Because  this  analysis  included  only  services 
provided  directly  by  the  FASST  program,  it  is  likely  that  the 
analysis  underestimates  total  service  utilization  across 
systems  to  which  the  child  and  family  may  have  been 
referred. 


O 

ERIC 


15S 


The  annual  cost  of  the  program  per  child  was  $5,300.  The 
Joint  Venture  FASST  funding  included  the  salaries  of  the 
Family  Coordinators,  Family  Advocates,  the  Program 
Manager,  and  the  Administrative  assistant.  It  also  included 
funds  to  access  a pool  of  clinic-based  therapists, 
home-based  therapists,  and  respite  workers  on  an  "as  needed 
basis"  consistent  with  the  families'  support  plans.  The 
annual  cost  of  services  per  child  is  overestimated,  since  the 
program  is  aimed  at  serving  the  entire  family  (i.e.,  family 
members  also  may  derive  benefit,  though  they  are  not 
necessarily  ("billable").  If  one  accounts  for  the  siblings  (an 
estimated  220  people),  and  caregivers  (conservatively 
estimated  at  one  per  household),  the  cost  falls  to  $1,281  per 
year. 

Outcomes 

Changes  in  functioning  at  home  and  in  school  were  assessed 
using  the  Parent  and  Teacher  Rating  forms  (FMHI,  1995a; 
FMHI,  1995b).  Results  suggest  that  parents  felt  that  their 
child's  behavior  had  improved  at  least  slightly  since  the 
onset  of  FASST  services.  Rated  on  a six-point  scale  from  no 
improvement  to  greatly  improved,  69%  of  parents  felt  their 
child  had  made  at  least  modest  gains  in  their  behavior  at 
home,  and  64%  reported  similar  gains  in  the  child's  ability 
to  get  along  with  other  family  members.  Fifty-six  percent  of 
parents  reported  modest  to  great  gains  in  the  child's 
interactions  with  peers,  and  62%  reported  at  least  modest 
gains  in  their  child's  self-esteem.  Teachers  also  noted 
improvements  in  child's  behavior  at  school,  academic 
performance,  self-esteem,  and  interactions  with  peers  (see 
Figure  2). 

Changes  in  attendance,  suspensions,  grades,  and  out  of 
home  placements  were  additional  measures  used  to  assess 
the  impact  of  the  FASST  services.  Examination  of  changes 
in  attendance  for  14  subjects  (N  = 14)  between  the  year 
prior  to  enrollment  in  FASST  and  the  first  year  of 
enrollment  show  a decrease  from  an  average  of  1 8. 1 days 
absent  to  14.9  days,  which  is  a positive  trend.  Suspension 
data  were  more  difficult  to  interpret,  because  differences 
may  be  a function  of  children  getting  older  as  well  as  impact 
of  the  program.  In  the  year  prior  to  enrollment  there  were  no 
documented  incidents  of  in-school  or  out-of-school 
suspension.  The  following  year,  when  children  were  first 
enrolled,  four  children  were  given  in-school  suspensions, 
and  1 8 children  were  suspended  out-of-school  for  an 
average  2.8  days  for  each  event.  Grade  data,  while  difficult 
to  standardize,  show  promise  as  well.  Between  the  year 
prior  to  enrollment  in  FASST  and  the  year  first  enrolled, 
46%  of  the  children  improved  in  at  least  one  math  or 
reading  area.  Improvements  were  more  marked  in  the  year 
first  enrolled  and  the  following  year,  in  which  91%  (N  = 21) 
improved  in  at  least  one  math  or  reading  area. 

Importantly,  children  enrolled  in  FASST  were  maintained  in 
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their  homes.  After  the  end  of  the  1995-6  school  year,  most 
children  (92%)  still  lived  at  home  with  their  natural  or 
extended  families.  Though  13%  of  the  children  had  been 
placed  out  of  home  in  either  foster  care  or  other  residential 
placement,  some  had  returned  to  their  homes  by  the  end  of 
the  reporting  period.  Finally,  parents  (N  = 48)  were  asked  to 
evaluate:  (a)  the  extent  to  which  their  family's  strengths 
were  considered  in  the  plaiming  process;  (b)  whether  they 
had  adequate  information  to  make  decisions  about  working 
with  the  FASST  team;  (c)  the  degree  of  participation  in 
developing  and  implementing  the  Family  Support  Plan;  and 
(d)  whether  their  opinions  were  valued,  respected,  and 
incorporated  in  the  plans.  Figure  3 shows  the  high  level  of 
satisfaction  in  these  particular  areas. 

There  are  obvious  shortcomings  to  measuring  the  outcomes 
(e.g.,  primarily  relatively  small  numbers  and  lack  of 
standardized  measures  of  functioning  and  achievement). 
These  issues  have  been  addressed  through  implementing 
standardized  measures  of  achievement,  symptoms,  and 
problem  behaviors,  as  well  as  home,  school,  and  community 
functioning  and  will  allow  for  more  precise  reporting  in  the 
future.  Nevertheless,  the  preliminary  attempts  to  document 
outcomes  show  promising  trends. 

Discussion 

In  the  process  of  developing  the  evaluation  plan,  collecting 
the  data,  and  interpretation,  there  were  several  issues 
highlighted  that  affect  continued  program  improvement. 
These  issues  can  be  summarized  as  caseload  issues  and  data 
as  well  as  evaluation  issues.  The  "heavy  users"  analysis  in 
which  it  was  discovered  that  one-quarter  of  children  and 
families  accounted  for  two-thirds  of  staff  time  led  to  two 
issues  which  are  now  a topic  of  program  planning.  The  first 
of  these  is  determining  the  optimal  caseload  mix.  This 
discussion  has  centered  around  balancing,  perhaps  through  a 
weighting  scheme,  "heavy"  versus  "regular"  users  of 
services,  so  that  all  children  and  families  receive  an 
appropriate  degree  of  service  intensity  and  staff  availability. 
The  second  issue  reflects  nationwide  debate  about  child 
versus  family-centered  services.  Upon  examining  the  costs 
per  child  and  per  family  member,  the  issue  of  "who  is  the 
client"  was  again  raised  (i.e.,  the  unit  of  service  question). 
While  philosophically  the  program  is  intended  to  meet  the 
family's  needs,  funding  streams  still  operate  on  an 
"identified  child"  model  and  staffs  efforts  are  then 
underestimated.  Another  caseload  issue  was  that  of 
continuity  of  care  and  the  wraparound  tenet,  unconditional 
care.  In  the  course  of  collecting  data,  the  high  mobility  of 
the  target  population  was  documented.  Children  and 
families  who  lived  in  the  catchment  area  of  the  six  schools 
at  the  start  of  the  program  had  moved  to  27  schools  within 
1 8 months.  While  the  program  is  committed  to  staying  with 
families,  the  dispersal  of  families  raises  logistical  and 
productivity  issues  for  staff  who  must  spend  a great  of  time 
traveling. 
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Data  and  evaluation  issues  also  were  raised.  Specifically,  it 
was  important  to  the  program  and  evaluation  staff  that  (a) 
data  elements  be  useful  at  the  individual  and  program  level, 
(b)  they  be  reasonably  easy  to  get  access  to,  and  (c)  they  be 
uniformly  collected  across  the  two  agencies  that  provide 
services.  The  first  criteria,  broad  utility  of  measures  led  to 
the  selection  of  the  CBCL,  TRP,  and  CAFAS.  For 
clinicians,  these  measures  produce  a profile  which  can 
provide  feedback  and  inform  the  family  support  planning 
process.  Program-wide,  these  measures  provide  a good 
description  of  who  the  children  are,  and  this  information  can 
be  compared  to  similar  programs  in  the  county.  For  the 
evaluation  staff,  these  instruments  are  well  documented 
psychometrically.  Access  to  the  data  elements  was  not 
consistent  and  led  to  both  the  reformulation  of  intake 
paperwork  and  an  on-going  discussion  of  how  to  integrate 
and  streamline  agency  and  school  data  requirements,  data 
required  by  the  funders,  and  data  for  use  in  the  evaluation. 
Finally,  in  compiling  service  use  data,  it  was  discovered  that 
although  the  two  agencies  had  the  same  accounting 
software,  billing  and  accounting  practices  differed  between 
them,  and  hence,  there  were  similar  but  not  directly 
comparable  figures  available  for  the  proportions  of  the 
various  services  provided. 
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Introduction 

Many  children  with  emotional/behavioral  disturbances  are 
adjudicated  dependent  due  to  abuse  and  neglect  and 
languish  in  the  foster  care  system  for  years,  frequently 
transferred  from  one  residential  placement  to  another,  with 
little  or  no  progress  toward  permanency  or  improved 
emotional/behavioral  adjustment.  To  explore  a possible 
solution  to  this  situation,  the  Fostering  Individualized 
Assistance  Program  (FIAP)  was  developed  to  provide 
individualized  wraparound  supports  and  services  to  foster 
children  with  emotional/behavioral  disturbances  and  to  their 
families  (i.e.,  foster,  biological,  and/or  adoptive).  Outcome 
findings  from  a controlled  study  suggest  that  the  FIAP 
intervention  was  somewhat  more  effective  than  standard 
foster  care  services  in  reducing  delinquency  and 
externalizing  behaviors  in  boys  and  in  significantly 
increasing  the  likelihood  of  permanency  living 
arrangements  for  older  youth. 
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Method 


The  children  who  were  included  in  this  study  were  in  foster 
homes  or  group  shelter  care  at  the  start  of  the  project  and 
had,  or  were  at  risk  of  having,  emotional/behavioral 
disturbances.  These  children,  ages  7 - 15  as  they  entered  the 
study,  had  been  out  of  their  homes  for  an  average  of  2.6 
years  and  were  changing  placements  at  an  average  rate  of 
four  times  per  year.  These  children  represented  the  most 
challenging  10%  of  the  foster  care  population,  having  been 
provided  few,  if  any,  mental  health  and  related  services 
within  the  dependency  system. 

The  FIAP  goals  for  its  children  and  families  were  to 
stabilize  child  placement,  improve  child  behavioral  and 
emotional  adjustment,  and  achieve  appropriate  permanency 
placements.  These  goals  were  facilitated  through  four 
clinical  components:  (a)  child  and  family  assessment  that 
addressed  individualized  strengths  and  needs;  (b) 
life-domain  service  planning  to  support  and  enhance 
permanency  plans;  (c)  clinical  case  management  of 
individualized,  wraparound  supports  and  services;  and  (d) 
follow-along  supports  to  maintain  permanency.  At  the  heart 
of  the  FIAP  intervention  were  family  specialists  who  served 
as  family-centered,  clinical  case  managers  and  home-based 
counselors,  collaborating  with  parents  and  other  family 
members,  foster  caseworkers,  other  providers  (e.g.,  teachers, 
therapists,  scout  leaders),  and  foster  parents.  The  family 
specialists  followed  and  served  their  children  across  all 
settings,  wrapping  services  around  them,  as  needed.  Our 
recommendations  for  individualized,  family-focused 
practices  have  been  published;  however,  it  is  important  for 
the  reader  to  understand  that  these  recommended  practices 
evolved  over  the  course  of  this  study  (McDonald,  Boyd, 
Clark,  & Stewart,  1995). 

The  FIAP  intervention  was  evaluated  in  a controlled, 
random-assignment  study  that  compared  a sample  of  at-risk 
children  who  received  this  individualized,  wraparound 
process  (FIAP  group,  n = 54)  with  a comparable  sample  of 
children  who  experienced  practices  that  were  standard  in  the 
Florida  foster  care  system  (SP  group,  n = 77). 

Results 

The  major  outcome  results  suggest  that:  (a)  FIAP  children 
were  significantly  less  likely  to  change  placements  than 
were  those  in  the  SP  group  during  the  intervention  period; 
(b)  both  groups  showed  significant  improvement  in  their 
emotional/behavioral  adjustment  over  time;  (c)  FIAP  boys 
were  more  likely  to  show  significantly  lower  rates  of 
delinquency  and  better  externalizing  adjustment  than  their 
SP  counterparts;  and  (d)  the  older  FIAP  youth  were 
significantly  more  likely  than  SP  youth  to  be  in  permanency 
settings  with  their  parents,  relatives,  adoptive  parents,  or 
living  on  their  own.  The  only  statistically  significant 
differences  between  the  groups  regarding  school 
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differences  between  the  groups  regarding  school 
performance  were  that  extreme  numbers  of  days  absent  were 
lower  for  the  FIAP  youth  than  for  the  SP  youth,  and  extreme 
numbers  of  days  of  suspensions  were  lower  for  the  FIAP 
group  than  for  the  SP  group.  Examinations  of  other 
community  adjustment  indicators,  for  subsets  of  youth  who 
had  any  history  of  runaways  or  incarceration,  suggest  that 
the  older  FIAP  youth  spent,  on  average,  fewer  days  per  year 
on  runaway  or  incarceration  status  during  the  post  period 
than  did  the  older  SP  youth  (Clark,  Lee,  Prange,  & 
McDonald,  1996;  Clark  et  al.,  1994;  Clark  et  al.,  in  press). 

Discussion 

Implications  for  Children's  Systems  of  Care 

Through  this  grant-funded  research  effort,  FIAP  has 
developed  and  refined  an  intervention  strategy  for 
improving  the  extemalizing/delinquency  adjustment  of  boys 
and  the  permanency  placements  of  older  youth  with 
emotional  and  behavioral  challenges  who  have  been  out  of 
their  homes  for  extended  periods.  These  differential  results 
were  achieved  even  though  a new  governor  and  class  action 
law  suit  provided  new  resources  for  the  standard-practice 
foster  care  and  adoption  system  during  this  study.  It  appears 
that  the  FIAP  intervention  strategy  might  be  strengthened 
further  by  ensuring  greater  consistency  in  the  individualized 
wraparound  approach  through  more  systematic  use  of 
family  therapy  and  field  supervision  methods. 

FIAP  personnel  are  currently  disseminating  programmatic 
information,  and  providing  staff  training  and  technical 
assistance  to  improve  practices  in  communities  in  Florida 
and  in  other  states.  The  following  recommendations  may 
prove  helpful  in  the  reform  of  practices  and  policies 
regarding  child-serving  agencies  for  improving  the 
humanness  and  effectiveness  of  systems  serving  children  at 
risk  and  their  families. 

Recommendations 

Foster  care  and  mental  health  systems  should  maximize  the 
likelihood  of  children  remaining  in  their  homes  of  origin, 
assuming  their  safety  can  be  ensured,  through  the  use  of 
family  preservation  and  family  systems  therapy  (Henggeler 
et  al.,  1994).  For  those  children  who  are  at  risk  of  having,  or 
do  have,  emotional/behavioral  disturbances  and  face 
extended  stays  within  the  dependency  system  (Boyd, 
Struchen,  & Panacek-Howell,  1989;  1990),  the  FIAP 
Research  Demonstration  principals  have  formulated  the 
following  recommendations. 

1 . Implement  an  individualized  wraparound  intervention 
process  that  would  be  external  to,  but  collaborative  with,  the 
foster  care  and  mental  health  systems,  to  ensure  that 
children  with  severe  emotional  disturbances  who  have  been 
abused  achieve  appropriate  permanency  placements 
(McDonald  et  al.,  1995). 
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• Establish  a wraparound  team  for  each  child  composed 
of  key  players  in  his/her  life  (e.g.,  biological  parent, 
foster  parent,  teacher,  foster  counselor,  therapist,  aunt, 
and  family  specialist)  to  develop  and  modify  service 
plans,  monitor  service  provision  and  outcomes,  and 
track  progress  toward  permanency. 

• Use  family  specialists  (e.g.,  clinical  case  managers), 
empowered  by  the  wraparound  team,  to  provide  child- 
and  family-focused,  wraparound  services,  with  an 
outcome  priority  on  permanency. 

• Complete  comprehensive  assessments  with  the 
children  and  their  families  (e.g.,  natural,  foster, 
relative,  adoptive),  to  determine  their  strengths,  needs, 
and  clinical  issues.  This  assessment  information 
should  guide  the  wraparound  team  in  service  delivery 
and  permanency  planning. 

• Ensure  that  the  family  specialists  and  wraparound 
teams  operate  under  a value  of  unconditional 
commitment,  in  that  they  will  not  deny  services  to  a 
child,  but  rather  adjust  services  and  supports  to  meet 
the  changing  needs  of  children  and  their  family 
circumstances  (VanDenBerg  & Grealish,  1996). 

2.  Remove  all  incentives  for  not  providing  effective, 
individualized,  family  centered  care  and  treatment. 

• Do  not  allow  family  specialists  who  work  with  this 
population  (i.e.,  children  with,  or  at  great  risk  of 
having,  severe  emotional  disturbance)  to  have  more 
than  1 0 children  on  their  active  case  loads,  keeping  in 
mind  that  even  those  children  who  have  been  placed 
in  permanent  settings  will  continue  to  need  follow 
along  services.  The  active  caseload  of  10  is 
contingent  upon  the  family  specialists  not  having  to 
provide  all  of  the  in-home  and  out-of-home 
therapeutic  services.  They  should  be  in  a position  to 
broker  and  monitor  numerous  supports  and  services 
(e.g.,  home-based  behavioral  support  therapist,  after 
school  mentor,  family  systems  therapist). 

• Do  unconditionally  commit  to  the  development, 
implementation,  evaluation,  and  follow  along  of 
individualized,  family-focused  services.  Provide 
family  specialists  with  adequate  and  flexible  funds  so 
they  are  able  to  address  crucial  service  areas  for  both 
child  and  family  (e.g.,  arrange  sexual  abuse  therapy  to 
occur  immediately  for  victims;  purchase  of  a 
refrigerator  for  a mother  may  remove  the  remaining 
barrier  to  family  reunification).  Further  empower  the 
family  specialists  with  monitoring,  facilitating, 
trouble-shooting,  and  on-the-spot  decision-making 
authority  regarding  the  implementation  of  service 
plans  (to  be  confirmed,  or  later  modified,  by  the 
wraparound  team). 

• Protect  against  premature,  facile,  or  unsupported 
Termination  of  Parental  Rights,  by  careful. 


166 


Termination  of  Parental  Rights,  by  careful, 
strength-based  team  review  of  all  such  proposals. 
Similarly,  do  not  specify  a permanency  plan  without  a 
carefully  wrought,  outcome-oriented  service  plan 
attached. 

• Do  include  biological  families,  as  well  as  foster 
families,  in  all  planning  and  decision  making.  Provide 
the  wraparound  team  with  authority  to  determine  the 
service  and  permanency  plans  that  will  be  submitted 
to  the  foster  care  supervisor  and  presented  to  the 
court. 

• Provide  family  specialists,  who  have  professional 
training  and  experience  in  the  provision  of  culturally 
sensitive  individualized  services,  with  weekly  clinical 
supervision  of  their  case  loads  and  with  field 
supervision  on  a bi-monthly  basis.  Identify  additional 
professional  expertise  (e.g.,  family  systems  therapy, 
behavioral  support  intervention,  sexual  abuse  therapy) 
to  tap,  as  needed,  for  staff  training  and  consultation 
and/or  for  direct  family  services. 

3.  Link  permanent  parents  with  naturally  occurring  supports 
in  those  areas  of  need  which  are  crucial  to  successful 
permanency  maintenance  of  each  child  in  his/her  eventual 
home. 

4.  Work  specifically  toward  the  long-term  goal  of  having 
each  permanency  family  be  its  own  case  manager,  averting 
situations  that  could  cause  recidivism  or  the  need  for 
additional  services  in  order  to  retain  the  child  in  his/her 
permanent  home.  Be  certain  to  contact  each  permanent 
family  at  planned  intervals,  to  provide  support  where 
needed,  until  self-reliance  is  reasonably  predictable. 

5.  While  stringently  monitoring  each  child's  progress 
towards  permanency  placement,  allow  the  wraparound  team 
to  exceed  the  legislatively  mandated  time  constraints,  as 
needed  (e.g.,  maximum  of  1 8 months  in  foster  care). 

6.  Do  not  allow  any  more  movements  of  a child  from  foster 
home  to  foster  home,  and  school  to  school,  than  absolutely 
necessary. 

7.  Use  level  of  severity,  age,  sex,  and  number  of  children  as 
essential  determinants  of  where  each  child  is  placed, 
avoiding  inappropriate,  dangerous,  or  combustible  mixes 
(e.g.,  do  not  place  young,  naive,  children  with  older, 
street-wise  youth). 

8.  Advocate  with  school  staff  to  ensure  that  each  child  is 
receiving  all  appropriate  services  in  the  least  restrictive 
environment  possible.  Assist  in  linking  parent  and  teacher 
(guidance  counselor)  to  improve  the  possibility  of  a 
coordinated  set  of  services. 

9.  Advocate  for  the  foster  care  system  to  develop  and/or 
expand  its  use  of  treatment  foster  care  homes  (Chamberlain 
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& Reid,  1991;  Meadowcroft  & Trout,  1990;  Mikkelsen, 
Bereika,  & McKenzie,  1993). 
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Families  First  in  Essex  County:  A Family 
Designed  and  Implemented  System  of  Care 

Authors 

Introduction  Method  Results  Discussion 

Introduction 


The  goal  of  this  project  has  been  to  develop  a system  of 
services  for  families  with  children  with  emotional 
disturbance  that  is  designed  and  primarily  implemented  by 
consumers.  From  1993  to  1996,  the  project  was  primarily 
funded  by  a Department  of  Health  and  Human  Services 
Center  for  Mental  Health  Services  research  grant  received 
by  the  New  York  State  Office  of  Mental  Health.  The 
researchers  will  be  presenting  the  research  results  at  a future 
date.  This  summary  presents  a more  subjective  view  by  the 
founder  and  director  of  the  program. 

In  the  past  ten  years,  the  consumer  movement  has  changed 
the  face  of  service  delivery,  emphasizing  the  need  for  voice, 
ownership,  and  options  for  people  receiving  services,  and  an 
end  to  what  our  program  refers  to  as  "blame  and  shame." 
Value  has  been  placed  on  families  and  professionals 
working  in  a partnership  with  the  family  in  charge.  Despite 
these  values,  even  the  best  of  systems  have  been  primarily 
designed  by  professionals.  The  following  are  questions 
posed  by  this  endeavor: 

• What  would  a program  look  like  that  was  developed 
in  response  to  what  families  said  they  wanted  and 
needed? 

• Is  it  possible  to  develop  a responsive  system  in  an 
extremely  rural  area? 
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Method 


In  1991  several  small  grants  were  received  to  fund  a 
planning  year.  In-depth,  in-home  interviews  were  held  with 
24  families  who  had  children  with  serious  emotional 
disturbance  and  tape  recorded  for  subsequent  analysis.  The 
author  asked  each  family  the  following  questions: 

• What  was  most  helpful  to  you? 

• What  was  not  helpftil  to  you? 

• If  you  had  a magic  wand,  what  services  would  you 
wish  for? 

After  the  completion  of  the  interviews,  a Parent  Planning 
Committee  was  formed,  chaired  by  the  project  director.  The 
committee's  assignment  was  to  make  recommendations  for  a 
program  based  on  the  interview  material.  The  committee 
met  for  eight  two-hour  sessions,  which  were  videotaped. 
During  the  course  of  this  planning  year,  an  Essex  County 
Child  and  Family  Task  Force  was  formed,  which  included 
service  providers,  community  people,  and  consumers.  The 
task  force  prioritized  service  system  needs  and  were  trained 
in  Child  and  Adolescent  Service  System  Program  (CASSP) 
principles,  with  emphasis  on  family  centered  services.  At 
each  meeting,  they  heard  a speaker  who  was  a consumer  tell 
the  story  about  their  experience  as  a service  recipient. 

Results 

There  was  almost  complete  unanimity  as  to  what  families 
reported  they  needed  and  wanted.  The  following  needs  were 
expressed  by  the  families  interviewed;  respite;  information 
and  referral;  an  advocate;  community  friend/mentor  for  their 
child;  support  for  caregivers,  siblings,  and  the  identified 
child;  crisis  services;  concrete  assistance;  and  family  center. 
When  professionals  design  services  for  families,  the 
services  that  were  prioritized  by  families  are  usually 
developed  years  after  other  more  traditional  services,  if  at 
all. 

The  following  is  a sample  of  the  32  recommendations  made 
by  the  Parent  Planning  Committee  and  implemented  by 
Families  First  in  Essex  County:  (a)  Preference  should  be 
given  to  parents  of  children  with  special  needs  in  hiring;  (b) 
participants  should  be  included  in  all  trainings,  as  both 
attendees  and  trainers;  (c)  no  interagency  meetings  should 
be  held  about  families  unless  they  are  present  (i.e.,  "Nothing 
About  Me  Without  Me");  (d)  there  should  not  be  any 
criteria  for  receiving  services  other  than  a family's  assertion 
that  their  child  has  emotional/behavioral  problems  and  they 
would  like  help;  and  (e)  there  should  not  be  any  waiting  list 
(i.e.,  the  program  should  be  immediately  responsive  by 
providing  support  services). 

More  intensive  services  such  as  case  management  are  not 
available  to  all  referrals  to  Families  First,  but  everyone  who 
calls  for  assistance  can  utilize  a variety  of  services.  Families 
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calls  for  assistance  can  utilize  a variety  of  services.  Families 
may  borrow  books  or  videotapes  from  the  Resource  Library. 
They  may  call  the  800  number  for  support  from  a 
parent/professional,  or  they  drop  in  to  the  Center  for  ■ 

support.  Families  are  provided  social  events  to  attend.  They 
can  receive  the  family  written  newsletter  and  notices  of 
events.  Families  may  request  an  initial  home  visit  for 
planning,  or  even  ask  to  be  matched  with  another  participant 
for  support.  Also,  they  can  receive  food  from  the  Families 
First  food  shelf  and  concrete  help  through  the  flexible  dollar 
fund.  Families  have  access  to  both  respite  for  child  care  and 
the  respitality  program  for  an  overnight  stay  in  a hotel. 
Finally,  they  may  have  an  opportunity  to  assist  another 
family  and/or  contribute  in  other  ways  to  the  organization. 

The  Parent  Planning  Committee  felt  strongly  that  the 
language  that  professionals  use  can  connote  either  respect  or 
blame  and  shame.  They  recommended  that  the  following 
terms  be  used:  (a)  "Multi-stressed"  rather  than 
"dysfunctional;"  (b)  "participant"  rather  than  "client;"  (c) 
"family"  rather  than  "case;"  (d)  "family  advocate"  rather 
than  "case  manager;"  and  (e)  "cautious"  rather  than 
"resistant." 

The  following  summarizes  the  salient  learnings  from  this 
demonstration  project; 

• If  services  meet  people's  needs,  recognize  their 
strengths,  and  are  offered  in  a warm,  hospitable  and 
respectful  environment,  families  will  utilize  the 
service.  Even  those  who  are  usually  the  most  difficult 
to  engage  (e.g.,  low  socioeconomic  families  and 
mothers  who  are  depressed),  will  participate. 

• Traditional  service  providers  are  highly  invested  in 
maintaining  the  status  quo  and  are  likely  to  be  very 
cautious  about  accepting  a new  paradigm  of  family 
centered  service.  Although  significant  changes  were 
made  in  Essex  County,  such  as  having  consumer 
representatives  on  interagency  committees,  it  took 
three  years  before  most  other  providers  became 
comfortable  with  family  centered  values  and 
procedures.  Many  continue  to  be  very  cautious. 

• Hiring  consumers  is  essential  to  a family  centered 
service,  but  it  presents  many  challenges.  When  staff 
both  deliver  and  receive  service,  it  requires  great 
flexibility  and  sensitivity. 

• Creating  truly  individualized,  flexible  services  that 
incorporate  the  natural  support  system  requires 
comfort  with  ambiguity  and  complexity,  which  can  be 
an  administrative  and  bookkeeping  challenge. 

• There  is  a tendency  to  revert  to  old  paradigms.  In 
order  to  assure  that  a program  remains  "family 
friendly,"  providers  need  to  always  be  vigilant  against 
slippage. 

Discussion 

Families  First  in  Essex  County  opened  its  doors  in 
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November,  1992.  In  39  months,  it  has  served  250  families, 
and  the  county  has  had  fewer  hospitalizations  and  out  of 
home  placements.  An  evaluation  survey  of  consumers 
yielded  a very  high  degree  of  satisfaction.  Family  members 
have  said  "Families  First  is  my  family,"  and  "I  have  been 
lifted  out  of  my  depression  since  I joined  the  family  of 
Families  First."  It  has  been  clear  that  the  agency  has 
established  a sense  of  community.  The  majority  of 
participants  assist  other  families,  work  in  the  office,  help 
with  social  events  or  serve  on  the  Board  or  on  Advisory 
Committees. 

The  project  has  demonstrated  that  it  is  possible  to  develop  a 
system  of  services  that  is  designed  and  implemented  by 
families,  and  that  the  resulting  program  will  have  services 
that  actually  please  and  delight  recipients.  It  has  also 
demonstrated  that  a system  of  family  centered  support 
services  can  function  in  a very  rural  area  despite  the 
challenges  arising  from  lack  of  transportation,  isolation,  and 
a pool  of  trained  professional  people. 
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Introduction 


Families  with  children  who  have  emotional  or  behavioral 
disorders  face  complex  and  multiple  challenges.  Progress 
has  been  made  to  develop  child-centered  and  family-focused 
systems  of  support  to  help  families  cope  with  stressful 
circumstances  and  access  services  from  the  categorical 
systems.  However,  parent  input  is  rarely  sought  when 
planning,  implementing,  and  evaluating  these  efforts. 

This  study  examined  the  nature  and  extent  of  support 
families  received  from  their  informal  social  networks  and 
from  paid  professionals,  and  how  helpful  these  types  of 
assistance  were  in  meeting  child  and  family  needs.  Results 
indicated  that  more  support  was  provided  by  formal 
organizations  and  paid  professionals  than  by  informal 
organizations  and  unpaid  individuals.  Family  members, 
however,  provided  the  most  help/support  in  coping  with 

I daily  challenges.  In  addition,  the  greater  the  number  of 

functions  of  service  coordination  received,  the  more 
successful  parents  were  in  accessing  formal  support  and  the 
more  satisfied  they  were  with  their  family's  quality  of  life. 
Results  indicated  that  receipt  of  service  coordination  may 
contribute  to  positive  outcomes  for  families  with  children 
who  have  emotional  or  behavioral  disorders.  Findings  imply 
that  professionals  should  rely  more  on  parent  input  to  ensure 
that  community  support  systems  effectively  address  child 
and  family  needs. 
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Method 


Mail  survey  research  methods  were  used  to  explore  the 
nature  and  extent  of  support  families  received.  The 
following  areas  were  examined;  (a)  characteristics  of 
children  with  emotional  or  behavioral  disorders  and  their 
families,  (b)  types  of  informal  and  formal  support  families 
received,  (c)  relationships  between  child  and  family 
characteristics  and  support  received,  (d)  helpfulness  of 
support  received,  (e)  functions  of  service  coordination 
received,  (f)  the  relationship  between  receipt  of  functions  of 
service  coordination  and  parent  success  in  getting  help,  and 
(g)  the  relationship  between  receipt  of  functions  of  service 
coordination  and  child  well-being  and  family  quality  of  life. 

Participants 

Participants  were  parents  residing  in  Oregon  who  had  a 
child  with  an  emotional  or  behavioral  disorder.  A sample  of 
250  parents  was  randomly  selected  from  the  Oregon  Family 
Support  Network  (OFSN)  database  of  1000.  The  OFSN  is  a 
statewide,  parent-run,  information  and  family  advocacy 
organization  to  support  both  the  children  and  youth  with 
emotional,  behavioral,  and  mental  disabilities  and  their 
families. 

Survey  packets  were  mailed  to  all  selected  parents.  A total 
of  120  surveys  were  returned.  Of  these,  100  met  the 
requirements  for  inclusion  in  the  sample  to  be  used  for  data 
analysis.  The  40%  return  rate  of  useable  surveys  was 
sufficient  to  answer  the  research  questions. 

Instrument 

A self-administered  questionnaire  was  used  for  data 
collection.  The  instrument  was  pilot  tested  by  meeting  with 
three  parent-run  support  groups.  A total  of  1 8 parents, 
representing  (a)  rural  and  urban  settings,  (b)  a range  of  ages, 
incomes,  and  educational  attainment,  and  (c)  with  children 
of  different  ages  and  disability  labels,  attended  three 
separate  meetings.  As  a result  of  the  pilot  test,  minor 
changes  were  made  in  both  the  terminology  and  instructions 
for  completing  the  survey. 

Reliability 

The  survey  was  divided  into  five  parts.  A test-retest 
reliability  procedure  was  conducted  by  mailing  each  of  the 
four  parts  of  the  questionnaire  that  included  quantifiable 
responses  to  four  subgroups  of  the  1 00  respondents 
approximately  one  month  after  receipt  of  the  original 
questionnaire.  Absolute  agreement  was  calculated  for  Part  I 
(family  characteristics)  and  Part  II  (child  characteristics). 

For  Part  I the  agreement  ratio  was  89%,  and  for  Part  II  the 
agreement  ratio  was  82%.  For  Part  III  (support  families 
received),  the  percent  of  agreement  was  77%  and  for  Part  IV 
(family  service  coordination  and  family  outcomes),  the 
percent  of  agreement  was  87%.  Pearson  Correlation 
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percent  of  agreement  was  87%.  Pearson  Correlation 
Coefficients  were  calculated  for  items  with  interval  data  in 
Part  III  and  Part  IV.  The  highest  reliability  coefficient  was 
.90  for  responses  related  to  family  service  coordination.  The 
lowest  was  .73  for  the  items  related  to  helpfulness  of 
support  received.  Inter-rater  reliability  for  Part  V,  the  one 
open-ended  question,  was  calculated  for  1 0 of  the  65  parents 
who  responded.  The  average  agreement  between  raters  was 
85%. 


Analysis 

Descriptive  statistics  (frequencies,  percentages,  means,  and 
standard  deviations)  and  statistical  analyses  were  conducted 
using  the  Statistical  Program  for  the  Social  Sciences 
(SPSS).  When  testing  statistical  significance,  a conservative 
alpha  level  of  p < .01  was  used  to  protect  against  experiment 
wise  error  for  t-test  results,  chi-square  test  of  association, 
Pearson  Correlation  Coefficients,  and  Friedman  Two-Way 
Analysis  of  Variance  by  Ranks. 

Results 

Respondents  represented  families  residing  in  22  of  Oregon's 
36  counties,  characterizing  the  general  population  of 
Oregon.  The  children  and  youth  with  emotional  or 
behavioral  disorders  reflected  the  racial  mix  of  the  national 
population,  with  a slightly  lower  percentage  (87%)  reported 
as  Caucasian  than  in  the  Oregon  population.  A wide  range 
of  child  and  youth  disability  categories  were  represented. 
The  range  of  disability  labels  per  child  was  1 to  9,  and  a 
mean  of  2.6  per  child.  Of  the  100  children  and  youth 
represented  in  this  study,  82  (82%)  were  participating 
in-school  programs,  with  the  largest  number  (72%) 
attending  in-school  programs  that  were  not  associated  with 
day  or  residential  treatment. 

Parents  were  asked  to  report  the  extent  to  which  their 
families  received  the  functions  of  service  coordination 
during  the  past  year.  These  functions  were:  (a)  assess  the 
needs  and  strengths  of  the  child  and  family;  (b)  develop  the 
family  service  plan;  (c)  link  the  child  and  family  with 
services  appropriate  to  child  and  family  needs;  (d)  monitor 
the  delivery  of  services  and  child  and  family  progress 
toward  goals;  (e)  advocate  for  the  child  and  family;  (f) 
provide  information  to  parents  regarding  where  to  find 
resources;  and  (g)  teach  self-advocacy. 

Of  the  100  respondents,  29  (29%)  reported  receiving  none 
of  the  7 functions  of  service  coordination.  The  assessment 
function  was  received  most  frequently  (n  = 59).  The  least 
frequently  received  function  was  self-advocacy  (n  = 14). 

The  pattern  that  emerged  was  that  as  the  number  of 
functions  received  increased,  the  number  of  families 
receiving  those  functions  decreased.  School  personnel 
provided  service  coordination  more  frequently  than  any 
other  discipline  (n  = 29)  and  schools  combined  with 
education  service  districts  accounted  for  44%  of  the  service 


178 


education  service  districts  accounted  for  44%  of  the  service 
providers  identified  as  delivering  functions  of  service 
coordination. 

Families  with  children  ages  1 1 to  18  received  more 
functions  of  service  coordination  than  those  ages  3-10  and 
19  - 28;  the  group  of  1 1 families  with  young  adults  ages  19 
to  28  with  emotional  or  behavioral  disorders  received  less 
service  coordination  than  the  other  age  groups. 

To  examine  the  relationship  between  the  extent  of  service 
coordination  families  received  and  overall  parent 
satisfaction  with  family  quality  of  life,  satisfaction  was 
measured  on  a scale  of  1 to  4,  with  1 being  "not  at  all 
satisfied"  and  4 being  "very  satisfied."  The  mean 
satisfaction  rating  for  91  parents  was  2.19  (SD  = .9).  The 
Pearson  Correlation  Coefficient  of  r = .2675  was 
statistically  significant  at  the  p < .01  level  of  significance, 
indicating  a positive  relationship  between  parent  satisfaction 
with  family  quality  of  life  and  number  of  functions  of 
service  coordination  received.  Statistical  analysis  of  the 
relationship  between  parent  success  in  being  able  to  get  the 
support  their  child  and  family  needed  and  the  receipt  of 
functions  of  service  revealed  a significant  positive 
relationship  between  these  two  variables  (r  = .3387;  p < 

.0 1 ).  In  addition,  a correlation  of  r = .6 1 80  (p  < .00 1 ) 
indicated  a strong  positive  relationship  between  parent 
success  in  getting  help  and  family  quality  of  life. 

The  open-ended  question  asked  parents  to  provide 
additional  information  about  the  support  their  child  and 
family  received.  As  a result  of  content  analysis,  1 1 themes 
emerged.  Of  these,  three  were  considered  the  most  relevant 
to  the  purpose  of  the  study:  (a)  parent  feelings,  (b)  parent 
needs,  and  (c)  parent  recommendations.  Parents'  expressions 
of  stress  and  difficulty  in  coping  were  described  more 
frequently  than  feelings  of  satisfaction  and  hopefulness. 
Parent  needs  were  categorized  in  7 primary  categories. 

These  were  (a)  respite  care,  (b)  transition  services,  (c) 
services  for  young  adults  (independent  living:  vocational, 
life  skills,  recreational),  (d)  service  coordination,  (e) 
financial  assistance  for  health  services,  (f)  child's  behavior 
to  be  more  positive,  and  (g)  support  for  relinquishing 
custody.  Recommendations  made  by  parents  included 
increased  flexibility  and  coordination  of  services  across  the 
categorical  systems. 

Discussion 

This  study  explored  the  nature  and  extent  of  the  informal 
and  formal  support  families  receive  and  how  helpful  these 
types  of  assistance  were  in  meeting  the  needs  of  both 
children  who  have  emotional  or  behavioral  disorders  and 
their  families.  The  socioeconomic  characteristics  of  the 
families  reflected  the  general  population  of  Oregon.  Child 
characteristics  were  consistent  with  the  literature  describing 
the  characteristics  and  educational  placements  of  children 
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and  youth  with  emotional  or  behavioral  disorders.  However,  ! 
findings  may  not  be  generalizable  to  the  population  of 
children,  youth,  and  families  since  the  sample  was 
purposeful. 

Families  received  support  from  both  formal  and  informal 
sources.  However,  parents  relied  mostly  on  family  members 
(informal  support)  when  they  needed  someone  to  talk  to 
about  the  daily  challenges  they  faced.  In  addition,  even 
though  support  received  was  generally  perceived  as  helpful, 
parent  anecdotal  comments  suggested  that  the  current 
service  system  continues  to  be  fragmented  and  difficult  to 
access. 

Findings  from  this  study  support  continued  efforts  to 
provide  service  coordination  within  a child-centered  and 
family-focused  system  of  support.  Statistically  significant 
relationships  were  found  between  parent  success  in  getting 
needed  help  and  the  number  of  functions  of  service 
coordination  received.  These  findings  suggest  that  family 
service  coordination  may  be  an  effective  means  to  ensure 
that  families  receive  timely  and  appropriate  assistance. 

Results  from  this  study  support  future  research  that 
examines  community  service  delivery  programs  that 
systematically  incorporate  informal  social  networks  into 
community-based  service  delivery  in  order  to  increase  the 
amoimt  and  availability  of  support,  and  also  more  closely 
match  child  and  family  needs  with  the  resources  most 
appropriate  to  meet  those  needs.  This  concept  appears  more 
promising  as  local  communities  look  for  innovative  ways  to 
transform  the  categorical  systems  at  a time  of  dwindling 
resources  and  increased  demand  for  government 
accountability. 
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Introduction 


Mental  health  services  for  children  have  been  described  as 
inaccessible,  inappropriate  (Saxe,  Cross  & Silverman, 
1988),  fragmented,  duplicated,  too  restrictive,  not 
community  based,  being  driven  by  the  needs  of  the 
providers  or  payers  rather  than  by  the  needs  of  children  and 
their  families,  and  for  failing  to  include  parents  as  part  of 
the  treatment  team  (Knitzer,  1982;  Young,  1990).  With  a 
history  of  inadequate  services,  the  need  exists  to  assure  that 
appropriate  treatment  is  available  for  children  with  mental 
health  problems  and  their  families. 


The  purpose  of  this  study  was  to  gain  information  about 
parents'  perceptions  of  and  satisfaction  with  mental  health 
services  for  their  children.  Parents'  satisfaction  is  an 
important  outcome  measure  that  serves  as  an  objective  of 
service  delivery  and  a factor  in  improving  future  services. 
Parent's  level  of  satisfaction  was  defined  as  the  degree  that 
the  parent  was  pleased  with  parent-staff  interactions. 
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Method 

Research  Questions  and  Data  Analysis 

This  was  a cross-sectional,  descriptive  study  that  explored 
relationships  among  variables  at  one  point  in  time. 

The  following  research  questions  were  developed: 

1 . How  do  parents  describe  their  situation  of  having  a child 
with  a mental  health  problem? 

2.  What  services  do  parents  desire? 

3.  How  do  parents  expect  to  be  treated  by  staff? 

4.  What  are  parents'  perceptions  of  staff? 

5.  What  was  parents'  level  of  satisfaction  with  staff? 

6.  What  variables  are  associated  with  parents'  satisfaction? 

7.  What  variables  are  associated  with  child's  current  living 
situation? 

Descriptive  statistics,  frequencies,  and  themes  were  utilized 
for  the  first  five  questions.  T-tests  and  correlations  were 
used  to  analyze  the  association  of  variables  with  the 
dependent  variable,  parents'  satisfaction.  T-tests  were  also 
used  to  analyze  the  association  of  variables  with  the 
dependent  variable,  child's  current  living  situation. 

Subjects 

The  participants  in  this  study  were  parents  of  students  with 
mental  health  problems  who  were  part  of  an  Alternative 
Residential  Services  (ARS)  program  with  state  special 
education  fiscal  support.  Students  receiving  ARS  who  were 
9 years  of  age  or  older  were  also  invited  to  participate  in  the 
study  along  with  their  parents.  A total  of  160  mailings  were 
sent  to  family  units.  At  least  one  response  came  from  47 
family  units  (30%  response  rate),  including  46  mothers,  24 
fathers,  and  25  students. 

Students 

Students  were  primarily  boys  (84.8%)  and  were  between  the 
ages  of  7 and  20  years  (M  = 14.4  years).  Parents  reported 
that  students  had  mental  health  or  behavioral  problems  for  3 
to  20  years  (M  = 1 1 .2  years).  Both  mothers  and  fathers 
similarly  rated  the  seriousness  of  the  students'  mental  health 
or  behavior  problems  as  3.9  on  a five-point  numeric  scale 
with  5 being  "a  serious  problem"  and  1 being  "no  problem." 

Children's  placements  for  services  were  42.6%  at  home, 
36.2%  in  in-state  residential  settings,  and  12.8%  in 
out-of-state  settings.  Another  8.5%  were  in  other  settings 
such  as  a nursing  home  and  a supervised,  independent  living 
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such  as  a nursing  home  and  a supervised,  independent  living 
situation.  The  average  per  child  expenditure  of  ARS  funds 
was  $53,019.26  for  the  1994-95  academic  year,  with  a range 
of  $0  to  $171,232  annually. 

Instrumentation 

Parents  described  their  situation  with:  (a)  the  Parent's 
Stressors  Scale,  (b)  the  Cause  of  my  Child's  Problem  Scale 
(CMCP),  and  (c)  the  Burden  Assessment  Scale  (BAS). 

The  Parent's  Stressors  Scale.  Parents  identified  and 
prioritized  the  stressors  they  experienced  as  parents  of 
children  with  mental  health  problems. 

The  Cause  of  My  Child's  Problem  Scale  (CMCP).  The 
18-item  CMCP,  a revision  of  McCauley's  (1992)  Causal 
Dimension  Scale  II,  measured  six  causal  dimensions 
including;  (a)  locus  of  causality  within  or  external  to  the 
parent;  (b)  stability;  (c)  personal  control;  (d)  external 
control;  (e)  locus  of  causality  within  or  external  to  the  child; 
and  (f)  pervasiveness.  The  coefficient  alpha  for  the  current 
study  was  .65  for  mothers  and  .64  for  fathers. 

The  Burden  Assessment  Scale  (BAS).  The  19  item  BAS,  a 
revision  of  Rinehard's  BAS  (1994),  measured  parents' 
perceived  burden,  including  objective  and  subjective 
burden.  The  coefficient  alpha  in  the  current  study  was  .93 
for  mothers  and  .91  for  fathers. 

Three  other  factors  related  to  parent  satisfaction  were 
measured,  including: 

• The  Parent  Satisfaction  Scale  (PSS).  The  7-item  PSS 
measured  parents'  level  of  satisfaction  with  their 
interactions  with  staff  (Gerkensmeyer,  1996). 
Coefficient  alpha  in  the  current  study  was  .91  for 
mothers,  .87  for  fathers,  and  .87  for  the  child  version. 

• The  Parent-Staff  Interaction  Scale  (P-SIS).  The 

13 -item  P-SIS  measured  parents'  perceptions  of  how 
they  were  treated  by  staff.  (Gerkensmeyer,  1996). 
Coefficient  alpha  in  the  current  study  was  .94  for 
mothers,  .87  for  fathers,  and  .87  for  the  child  version. 

• The  Expectation  of  Staff  Interactions  Scale  (ESI).  The 
5-item  ESI,  developed  by  the  author,  measured 
expectations  the  parent  or  child  had  about  staff. 
Coefficient  alpha  in  the  current  study  was  .89  for 
mothers,  .72  for  fathers,  and  .89  for  the  child  version. 

Results 

How  Parents  Described  Their  Situation 

Parents'  Stressors.  The  weighted,  prioritized  stressors 
identified  by  the  parents  included:  (a)  child  focused 
stressors  obtaining  406  points  (32%);  (b)  parental  role 
stressors  obtaining  356  points  (29%);  (c)  service  system 
stressors  obtaining  342  points  (27%);  and  (d)  family  focused 
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stressors  obtaining  342  points  (27%);  and  (d)  family  focused 
stressors  obtaining  162  points  (13%). 

The  Cause  of  My  Child's  Problem  (CMCP).  Parents  did  not 
believe  they  had  personal  responsibility  for  causing  their 
child's  mental  health  problem.  The  CMCP  scale  least 
endorsed  by  parents  on  the  CMCP  was  the  parents'  locus  of 
causality  subscale  (i.e.,  within  you  as  a parent;  M = 3.8  on  a 
nine-point  scale). 

Overall,  parents  tended  to  explain  their  children's  mental 
health  problems  as  pervasive,  likely  to  last  forever,  and  not 
caused  by  themselves.  The  most  highly  endorsed  CMCP 
items  by  mothers  were  will  affect  all  parts  of  your  life  and  is 
inside  your  child  (M  = 7.1),  followed  by  "will  last  forever" 
(M  = 7.0).  For  fathers,  the  most  highly  endorsed  item  was 
will  last  forever  (M  = 7.2),  followed  by  is  inside  your  child 
(M  = 6.7). 


Burden  Assessment.  The  parents'  most  highly  endorsed 
BAS  item  was  worried  about  what  the  future  holds  for  your 
child  (M  = 3.7  on  a 4 point  scale).  The  least  endorsed 
burden  for  both  mothers  and  fathers  was  resented  your  child 
because  s/he  made  too  many  demands  on  you  (M  = 2.0  and 
1 .9,  respectively). 

Parents'  Desired  Services 

Several  parents  (48%)  indicated  that  they  wanted  exactly 
what  services  their  children  were  receiving.  Parents  also 
endorsed  that  they  wanted  respite  services  (34%),  school 
based  services  (27%),  in  state  residential  services  (27%), 
home  based  services  (18%),  out  patient  counseling  (15%), 
in-state  hospitalization  (8%),  and  partial  hospitalization 
(5%).  None  of  the  parents  endorsed  wanting  out  of  state 
services;  however,  50%  (6  out  of  12)  of  those  receiving 
out-of-state  services  endorsed  wanting  exactly  what  they 
were  receiving. 

Expectations  of  Staff  Interactions 

Mothers  and  fathers  most  strongly  agreed  that  they  expected 
to  be  treated  well  by  staff.  Parents  least  agreed  that  staff 
treated  them  better  than  they  expected. 

Perceptions  of  Staff  Interactions 

In  general,  parents  had  highly  positive  perceptions  of,  and 
satisfaction  with,  their  interactions  with  staff.  The  most 
negative  perceptions  were  within  a neutral  range.  Positively 
skewed  data  have  been  a consistent,  long  standing  problem 
with  consumer  satisfaction  and  perception  data  (Lebow, 
1982),  therefore,  in  identifying  areas  to  improve,  the  least 
positive  responses  need  to  be  targeted,  even  if,  as  in  this 
study,  they  fall  in  a neutral  range.  Mothers  and  fathers  did 
not  describe  the  staff  as  rude  to  them  (M  = 2.2  and  2.0, 
respectively  on  a five-point  scale  with  1 = strongly  agreeing 
and  5 = strongly  disagreeing  with  positive  perceptions  of 
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and  5 = strongly  disagreeing  with  positive  perceptions  of 
staff  interactions).  The  most  negatively  endorsed 
perceptions  for  mothers  were  that:  (a)  staff  were  (not) 
helpful  in  identifying  community  resources  (M  = 3.0),  (b) 
staff  had  limited  skills  to  help  their  children  (M  = 3.0),  and 
(c)  staff  (did  not)  ask  their  opinions  about  what  help  their 
families  needed  (M  = 2.9).  The  most  negatively  endorsed 
perceptions  for  fathers  were  that:  (a)  staff  were  (not)  helpful 
in  identifying  community  resources  (M  = 2.9),  (b)  staff  (did 
not)  fit  services  to  meet  the  needs  of  their  children  (M  = 
2.9),  and  (c)  staff  were  (not)  very  supportive  when  they 
were  in  distress  (M  = 2.8). 

Level  of  Satisfaction 

Mothers  and  fathers  were  least  satisfied  with  how  staff 
helped  them  find  services  their  children  needed  (M  = 3.0). 
They  were  most  satisfied  with  how  the  staff  treated  them 
with  respect  (M  = 2.3)  with  the  lower  mean  representing 
greater  satisfaction. 

The  Influence  of  Child's  Current  Living  Situation 

The  child's  living  at  home  was  associated  with  significantly 
higher  levels  of  parents'  total  burden,  subjective  burden,  and 
fathers'  objective  burden.  Further,  of  the  children  living  out 
of  their  home,  those  living  out-of-state  had  significantly 
lower  levels  of  parents'  subjective  burden,  total  burden,  and 
mother's  objective  burden  than  those  with  in-state, 
residential  placement  (see  Table  1). 

Parents'  Satisfaction 

The  more  satisfied  mothers  were,  the  more  (a)  positive  their 
expectations,  (b)  helpful  they  found  ARS  services,  (c) 
positively  staff  interactions  were  perceived,  (d)  likely 
parents  were  to  recommend  or  return  to  the  staff  for 
services,  and  (e)  satisfied  fathers  and  children  were  with 
staff  interactions  (see  Table  2 for  correlations). 

The  more  satisfied  fathers  were,  the  more  (a)  positive 
mothers'  expectations,  (b)  likely  fathers  would  recommend 
or  return  to  staff  for  services,  (c)  positively  parents 
perceived  staff  interactions,  and  (d)  satisfied  mothers  were. 
Further,  the  more  satisfied  the  fathers,  the  less  pervasive 
mothers  viewed  the  child's  problem,  the  younger  the  child 
was,  and  the  shorter  length  of  time  the  child  had  a problem. 
Fathers  were  also  more  satisfied  when  the  child  was  placed 
out-of-state  and  least  satisfied  when  they  lived  at  home. 

Discussion 

With  a 30%  response  rate  and  small  sample  size,  the 
findings  must  be  interpreted  cautiously.  Nevertheless,  the 
findings  provide  some  information  concerning  the  parent's 
perceptions  of  their  experiences  with  mental  health  services. 
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Parents'  satisfaction  scores  suggest  that  parents  were 
generally  satisfied  with  the  interpersonal  interactions  of 
staff.  Provision  of  effective  case  management  is  indicated 
by  parents'  decreased  satisfaction  with  the  staffs  ability  to 
help  them  find  needed  services  for  their  children  and  the 
parents'  concern  for  their  children's  future. 

With  parents'  level  of  burden  significantly  related  to  their 
child's  current  living  situation,  provision  of  effective 
support  and  resources  is  indicated  in  order  to  assure 
families'  success  and  well-being  when  children  with  mental 
health  problems  are  living  at  home.  Parents'  most  highly 
endorsed  desired  service,  respite  service,  was  consistent 
with  their  need  for  support.  Furthermore,  services  need  to 
address  fathers'  needs,  as  fathers  had  significant  increases  in 
all  of  their  burden  scales  when  children  with  mental  health 
problems  lived  at  home. 

With  a significant  relationship  found  between  parents' 
reported  intent  to  return  or  refer  to  staff  and  their  level  of 
satisfaction,  parent  satisfaction  may  be  more  than  an 
objective  of  care  or  a factor  in  improving  services.  Parents' 
level  of  satisfaction  with  staff  interactions  may  also  be 
associated  with  subsequent  behavior  (e.g.,  continued 
engagement  with  the  service  delivery  system).  If  so,  parents' 
satisfaction  may  be  associated  with  other  functional 
outcomes  for  children  with  mental  health  problems.  Further 
research  is  indicated  to  substantiate  this  relationship. 
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Introduction 


Various  efforts  to  restructure  the  mental  health  service 
system  for  children  and  adolescents  have  identified  the 
value  of  a full  continuum  of  services,  coordinated  by  a case 
manager,  in  which  families  are  encouraged  to  actively 
participate  (Stroul  & Friedman,  1986).  Adequate 
assessments  of  the  effectiveness  of  such  "systems  of  care" 
require  accurate  information  on  service  utilization.  Very 
often,  utilization  data  are  obtained  from  a parent  of  the  child 
client.  This  research  examines  whether  parent  reports  of 
their  child's  service  use  are  more  reliable  in  a 
family-focused  delivery  system  than  in  more  typical  service 
environments  that  do  not  explicitly  embrace  "system  of 
care"  principles. 
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' Methods 

Sample 

Data  were  obtained  through  the  Fort  Bragg  Evaluation 
Project  (FBEP),  a longitudinal  assessment  of  a managed 
care  model  for  delivering  mental  health  services  to  children 
and  adolescents  (hereafter,  the  "Demonstration").  The 
present  research  is  based  on  about  600  youth  who  remained 
in  the  study  six  months  after  intake  to  the  Evaluation,  for 
whom  collateral  service-related  data  were  collected.!  About 
60%  of  this  sample  received  services  through  the 
family-focused  Demonstration,  and  40%  through  two 
traditional  systems  of  care  (hereafter,  the  "Comparison 
groups").  On  average,  the  sample  was  about  eleven  years 
old,  about  60%  male.  Seventy  percent  were  white,  20% 
were  African  American;  and  1 0%  were  of  other  or  mixed 
races.  About  80%  lived  in  a two-parent  household,  with  an 
annual  income  between  twenty  and  forty  thousand  dollars. 
Over  90%  of  the  youth  had  significant  functional 
impairment  (CAFAS;  Hodges,  1990);  over  97%  obtained  at 
least  one  clinical  diagnosis  based  on  parental  responses  to 
the  Child  Assessment  Schedule  (P-CAS;  Hodges,  Kline, 
Stern,  Cytryn,  & McKnew,  1982).2 

Data  Sources/Measures 

Utilization  data  were  obtained  from  the  youth's  primary 
caretaker  (i.e.,  usually  the  biological  mother)  and 
institutional  records  (i.e.,  the  management  information 
system  (MIS)  at  the  Demonstration) ;3  and  the  Civilian 
Health  and  Medical  Program  of  the  Uniformed  Services 
(CHAMPUS)  database  for  the  Comparison  groups.  The 
following  service  categories  are  examined:  a) 
hospitalization  (HOSP),  b)  residential  treatment  (RTC),  c) 
outpatient  therapy  (OUT),  and  d)  medical  evaluations 
(MEDEVAL).4  For  each  category,  two  measures  of 
utilization  are  analyzed:  service  use  (i.e.,  "yes"  or  "no")  and 
service  dose  (i.e.,  the  number  of  service  units  received). 

Data  are  based  on  reports  of  use  in  the  first  six  months  after 
intake.  Because  parents  may  miss  the  distinction  between 
hospitalization  and  residential  treatment,  or  between 
outpatient  care  and  medical  evaluations,  two  general 
categories  of  use  are  created.  "Restrictive"  care  includes 
hospitalization  or  RTC;  "nonrestrictive"  care  includes 
outpatient  care  or  medical  evaluations. 5 

Analyses 

Identical  analyses  of  reliability  were  performed  on  the 
Demonstration  and  Comparison  groups.  First,  the 
percentage  of  children  who  used  each  service  type  is 
reported  based  on  each  data  source.  Because  parent  and 
institutional  reports  can  reveal  identical  rates  of  use  without 
any  association  between  the  reports,  proportional 
agreements  between  the  two  sources  are  also  provided. 
Kappa  coefficients,  which  indicate  whether  the  agreement  is 
better  than  what  would  be  expected  by  chance  (Cohen, 
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better  than  what  would  be  expected  by  chance  (Cohen, 

1960)  are  shown.6  Agreement  can  arise  because  parents 
concur  with  institutional  reports  of  use  (i.e.,  "sensitivity")  or 
nonuse  (i.e.,  "specificity")  (Kraemer,  1992).  Therefore,  data 
are  also  shown  to  indicate  the  nature  of  agreement  between 
parent  and  institutional  reports.  Finally,  correlations 
between  collateral  reports  of  service  dose  are  shown. 

Results 

Table  1 shows  that  parents  in  the  Demonstration  and 
Comparison  groups  tend  to  report  roughly  equivalent  rates 
of  service  use  as  found  in  their  respective  institutional 
records.  Exceptions  include  parental  overreporting  of 
medical  evaluations  in  the  Demonstration  and  of  RTC  use 
among  the  Comparison  groups. 

Table  2 shows  high  percentages  of  agreement  between 
parent  and  institutional  reports  for  all  service  types,  in  both 
service  systems.  The  proportional  agreement  on  medical 
evaluations,  though  good,  is  the  lowest  observed.  Most 
notably.  Demonstration  families  indicate  much  better 
accuracy  than  the  Comparison  group  with  regard  to 
nonrestrictive  care,  specifically  for  outpatient  care.  The 
kappas  indicate  that  when  chance  agreement  is  eliminated, 
parents  in  both  systems  continue  to  report  reliable  use  of 
restrictive  care.  Low  kappas  for  nonrestrictive  services  are 
largely  due  to  the  relatively  inaccurate  reports  of  medical 
evaluations  and  the  large  number  of  youth  who  received 
outpatient  care. 7 In  short,  participants  in  the  family-focused 
system  of  care  provide  good  reports  overall,  better  than  their 
counterparts  in  traditional  systems  with  regard  to 
nonrestrictive  services,  though  no  better  with  regard  to 
restrictive  care. 

Figure  1 shows  that  reliability  depends  on  whether  one 
considers  the  sensitivity  or  specificity  of  reports. 8 
Regarding  restrictive  services,  the  strong  overall  agreement 
at  the  Demonstration  (in  Table  2)  is  largely  explained  by 
high  specificity  (98%).  Sensitivity  of  these  reports  is 
relatively  low  (69%),  with  parents  at  the  Demonstration 
tending  to  underreport.9  The  pattern  among  families  in 
traditional  systems  is  similar;  they  also  show  greater 
specificity  (94%)  than  sensitivity  (86%).  While  their 
specificity  rate  is  comparable  to  that  of  the  Demonstration, 
their  sensitivity  rate  is  higher. 

The  findings  with  regard  to  generalized,  nonrestrictive  care 
show  greater  sensitivity  than  specificity  in  both  types  of 
systems.  Outpatient  care  in  particular  is  overreported. 
However,  this  finding  must  be  interpreted  in  context  of  the 
number  of  clients  who  received  outpatient  care.  When  so 
many  receive  the  care,  only  a few  cases  incorrectly  reporting 
its  use  would  generate  a high  inaccuracy  rate.  10  The 
skewness  of  the  sample's  distribution  is  less  severe  in  the 
Comparison  group,  where  11%  did  not  receive  outpatient 
care.  Indeed,  perhaps  the  most  striking  finding  about  reports 
of  nonrestrictive  services  is  the  low  specificity  rate  (17%)  in 


191 


of  nonrestrictive  services  is  the  low  specificity  rate  (17%)  in 
the  traditional  systems.  Sensitivity  of  reports  of 
nonrestrictive  services  is  comparable  in  both  service 
systems.  The  low  reliability  of  medical  evaluation  reports  in 
both  systems  is  largely  attributable  to  low  sensitivity  (i.e., 
underreporting). 

Table  3 shows  that  parent  reports  on  doses,  as  with  use,  are 
more  accurate  for  restrictive  than  for  nonrestrictive  care.  In 
both  systems,  correlations  between  parent  and  institutional 
records  of  generalized  restrictive  care  are  high  and  the 
difference  between  systems  is  not  significant.  When 
hospitalization  is  examined  specifically,  however,  the 
correlation  at  the  Demonstration  (r  = .88)  is  significantly 
higher  than  in  the  Comparison  group  (r  =.31).  On  the  other 
hand,  correlations  at  the  Demonstration  for  nonrestrictive 
care  are  low,  and  significantly  lower  than  among  the 
Comparison  groups. 

Table  4 summarizes  the  findings.  A plus  sign  indicates  the 
system  with  the  stronger  association  between  parent  and 
institutional  reports;  a minus  sign,  the  system  with  the 
weaker  association.  Of  the  seven  specific  service  categories 
that  can  be  compared,  reports  for  the  two  service  systems 
differed  significantly  on  six.  Of  these  six,  four  favored  the 
Demonstration;  two,  the  Comparison  group.  The  general 
measures  indicate  that  the  accuracy  of  Demonstration 
families'  reports  of  use  and  dose  of  restrictive  care  is  about 
the  same  as  observed  among  families  in  the  traditional 
systems.  For  nonrestrictive  services.  Demonstration  families 
tend  to  be  more  accurate  with  regard  to  use,  but  less 
accurate  on  questions  about  the  number  of  service  units 
rendered. 

Discussion 

Efforts  to  provide  more  effective  mental  health  services  for 
children  have  embraced  the  value  of  including  children's 
families  in  the  treatment  process.  This  paper  explored  the 
possibility  that  parent  reports  may  be  more  reliable  in  a 
family-focused  system  than  in  traditional  systems  that  do 
not  explicitly  pursue  family-focused  interventions.  Findings 
suggest  that  participants  in  a family-centered  system 
generate  reliable  utilization  data.  However,  their  reliability 
does  not  consistently  exceed  that  observed  in  traditional 
environments.  With  regard  to  restrictive  care,  this  may  be 
due  to  the  introduction  of  innovative,  intermediate  services 
the  Demonstration  made  available.  Families  at  the 
Demonstration  showed  a greater  likelihood  of 
underreporting  restrictive  care  than  those  in  the  Comparison 
group,  a phenomenon  that  may  relate  to  the  greater  number 
of  additional  services  they  received  through  the 
Demonstration.  1 1 Reports  of  the  use  of  nonrestrictive  care 
are  good;  however,  reports  of  dose  are  not  very  reliable,  and 
significantly  more  unreliable  than  among  the  Comparison 
groups. 

The  Demonstration  represented  a new  and  rather  complex 
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The  Demonstration  represented  a new  and  rather  complex 
structure  with  which  parents  and  professionals  alike  had  to 
familiarize  themselves.  The  lack  of  consistent  superiority  in 
reliability  of  utilization  reports  through  the  Demonstration 
suggests  such  organizational  challenges  may  have  impeded 
the  system's  intended  capacity  to  meaningfully  include 
parents  in  treatment  planning.  Other  research  (Sonnichsen  & 
Heflinger,  1 993)  has  also  found  that  families  at  the 
Demonstration  were  not  significantly  more  involved  in 
restrictive  services  than  parents  at  the  Comparison  sites 
(though  they  were  more  involved  in  nonrestrictive  care). 
Together,  these  findings  suggest  that  we  may  need  to 
reorganize  or  redouble  our  efforts  to  include  families  as 
active,  knowledgeable  partners  in  the  therapeutic  process. 
Moreover,  it  may  be  that  any  added  value  of  family-focused 
service  systems  for  reliable  parent  reports  may  obtain  only 
after  participants  in  the  system  have  time  to  adapt  to  the 
new  organizational  environment. 
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Endnotes 


1 . The  follow-up  sample  used  for  this  research  is  more 
limited  than  in  the  original  Fort  Bragg  database  described 
elsewhere  (Bickman,  et  al.,  1995).  First,  about  10%  of  the 
Demonstration  sample  has  been  excluded  because  they 
"transitioned"  into  a different  level  of  care  at  the  time  they 
entered  the  evaluation.  These  parents  may  have  unusual 
difficulty  in  distinguishing  between  services  received  before 
and  after  transition,  particularly  when  these  services 
represent  treatment  for  the  same  episode.  Second,  about  100 
cases  at  the  Demonstration,  who  indicated  their  childhood 
an  initial  pretreatment  visit  but  did  not  return  for  treatment 
were  excluded  from  analyses.  Finally,  a small  number  of 
parents  in  the  Comparison  group  indicated  that  outpatient 
and  medical  evaluations  were  not  paid  by  CHAMPUS. 

These  cases  were  excluded  because  the  transactions  would 
not  be  included  in  the  CHAMPUS  database.  Thus,  the 
research  is  based  on  the  approximately  600  cases  for  which 
service  use  data  could  be  expected  in  both  the  parent  and 
institutional  database  for  the  same  six-month  interval. 

2.  The  child  and  family  profiles  of  the  Demonstration  and 
Comparison  groups  were  similar  on  factors  that  may 
differentially  affect  parent  recall  of  services,  for 
example&endashfthe  types  or  degree  of  their  child's 
problems  and  their  previous  experience  in  the  mental  health 
system.  See  Bickman,  et  al.,  1995  for  more  details  on  the 
sample  and  its  representation  of  other  youth  in  treatment. 

3.  Previous  analysis  compared  the  Demonstration's  MIS  and 
provider  data  and  found  strong  agreement  between  the  two 
sources  (Bickman,  et  al.,  1995).  Thus,  the  institutional 
record  is  considered  the  "gold  standard"  in  this  research,  to 
which  reliability  of  parent  reports  is  assessed. 

4.  The  Demonstration  offered  intermediate  levels  of  care 
(e.g.,  day  treatment)  that  were  unavailable  to  the 
Comparison  groups.  Only  the  four  service  types  offered 
through  both  service  systems  are  included  in  this  research. 

5.  Multiple  reports  of  use  of  the  same  service  type  within 
the  six  month  period  are  counted  once.  Dosages  associated 
with  multiple  reports  of  the  same  service  type  are  summed. 
Thus,  service  use  and  dose  represent  service  types  used  and 
the  total  dose  received,  regardless  of  the  number  of  episodes 
or  providers  associated  with  each  service. 

6.  Kappa  estimates  of  reliability  decrease  as  the  proportion 
of  cases  across  categories  departs  from  equality.  In  these 
data,  a sizable  majority  of  cases  is  classified 
similarly&endash;for  example,  more  than  90%  who  indicate 
use  or  nonuse  of  some  services.  In  this  situation,  the 
likelihood  of  chance  agreement  is  extremely  high  and  the 
ability  to  improve  prediction  beyond  chance  (as  indicated  by 
kappa)  is  weak.  Further,  estimates  of  reliability  also 
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decrease  as  the  number  of  response  categories  decreases. 
Thus,  kappas  will  be  lower  for  the  nominal  yes/no 
dichotomies  used  here  to  indicate  service  use  then  would  be 
expected  for  ordinal  measures.  Given  these  properties  of 
kappa  and  of  these  data,  kappas  should  be  interpreted  in 
conjunction  with  other  findings  presented. 

7.  It  is  difficult  to  improve  upon  chance,  thus  obtain  large 
kappas,  when  the  vast  majority  of  cases  fall  into  the  same 
category  which  is  the  case  here  with  outpatient  care. 

8 Less  sensitive  reports  suggest  that  parents  underreport 
service  use;  less  specific  reports  suggest  parents  overreport 
use.  Cases  (indicated  by  dots  for  the  Demonstration, 
triangles  for  the  Comparison  groups)  in  the  upper  right 
quadrant  of  Figure  1 suggest  relatively  high  specificity  and 
sensitivity.  Symbols  that  lie  below  or  to  the  left  of  this 
quadrant  suggest  diminishing  specificity  and  sensitivity, 
respectively. 

9.  At  the  Demonstration,  too  few  cases  had  reports  of  RTC 
use,  and  a sensitivity  measure  was  not  calculable.  Because 
both  sensitivity  and  specificity  measures  are  required  for 
charting.  Figure  1 does  not  include  an  indicator  ("dot")  of 
RTC  use  for  the  Demonstration. 

10.  The  proportion  of  clients  who  overreport  is  based  on  the 
total  number  of  clients  who  did  not  receive  that  type  of  care. 
For  the  Demonstration,  only  2%  (N  = 5)  of  families  did  not 
receive  outpatient  care.  Thus,  just  two  clients  incorrectly 
reported  use  yielding  a 40%  inaccuracy  rate. 

1 1.  Summerfelt,  Foster,  & Saunders,  (1966)  report  greater 
service  use  among  children  at  the  Demonstration  than  at  the 
Comparison  sites.  Further,  auxiliary  analyses  were 
conducted  on  Demonstration  clients  to  ascertain  whether 
their  underreporting  of  restrictive  services  was  associated 
with  overreporting  of  other  service  types.  Findings  showed 
that  underreporters  tended  to  report  a greater  number  of 
other  services  than  parents  who  accurately  reported  services, 
particularly,  intermediate  residential  services&endash;those 
that  parents  may  be  most  likely  to  confuse  with  traditional 
restrictive  settings.  And,  official  data  corroborated  the 
parents'  reports&endash;underreporters  did  in  fact  receive  a 
somewhat  greater  number  of  other  services  than  accurate 
reporters  (3.0  vs.  1 .5).  This  suggests  that  the  basis  for 
underreporting  may  be  due  to  confusion  that  understandably 
may  result  when  multiple  services  are  provided.  However, 
confidence  in  these  findings  is  dampened  by  the  small 
number  of  clients  at  the  Demonstration  who  received 
intermediate  services. 


Authors 

Carolyn  S.  Breda,  Ph.D. 

Research  Associate 

Center  for  Mental  Health  Policy 


O 

ERIC 


195 


Vanderbilt  University 
1207  ISthAve.  South 
Nashville,  TN  37240 
Voice:  615/322-8435 
Fax: 615/322-7049 
bredacs@ctrvax . vanderbilt . edu 

For  further  information,  contact  Dr.  Breda  at  Vanderbilt 
Center  for  Medical  Health  Policy,  1207  18th  Ave.  South, 
Nashville,  TN  37212.  This  research  was  supported  by  the 
U.S.  Army  Health  Services  Command 
(DA-DA10-89-C-0013)  as  a subcontract  from  the  North 
Carolina  Department  of  Human  Resources/Division  of 
Mental  Health,  Developmental  Disabilities  and  Substance 
Abuse  Services,  and  a grant  to  Dr.  Leonard  Bickman 
(ROlMH-46136-01)  from  the  National  Institute  of  Mental 
Health. 


Liberton,  “^^utasIi^^^nTieSmann^^^ES^^T^^T^Ti^tl^nnuan^^searcl^on^i^ce 
Proceedings,  A System  of  Care  for  Children's  Mental  Health:  Expanding  the  Research  Base  (February 
26  - 28,  1996).  Tampa,  FL:  University  of  South  Florida,  Louis  de  la  Parte  Florida  Mental  Health 
Institute,  Research  & Training  Center  for  Children's  Mental  Health. 


ERIC 


196 


The  9th  Annual  Research  Conference  Proceedings,  A System  of  Care  for  Children’s  Mental  Health:  Expanding  the 

Research  Base  (February  26  - 28,  1996). 


Introduction 


Parents  of  children  with  emotional  and  behavioral  disorders 
often  feel  that  they  are  not  treated  as  equal  partners  in  their 
children's  services  (Collins  & Collins,  1990).  Often,  they  are 
not  consulted  by  professionals  regarding  the  needs  of  the 
children  (Evans,  Armstrong,  Thompson,  & Lee,  1994). 
Parental  participation  in  children's  treatment  decisions, 
however,  has  begun  to  be  a mental  health  system  goal  that  is 
embraced  by  both  family  members  and  mental  health 
professionals  (DeChillo,  Koren,  & Schultze,  1 994;  Evans  et 
al.,  1994;  Friesen  & Korloff,  1990).  Indeed, 
parent-professional  collaboration  is  being  recognized  as  a 
component  of  professional  service  delivery  (Friesen  & 
Schultze,  1992;  Edelman,  Greenland,  & Mills,  1992). 

There  has  been  a lack  of  research  into  the  relationship  of 
collaboration  to  other  service  goals  such  as  satisfaction  and 
parent-professional  agreement  about  child  treatment 
priorities.  The  present  study  sought  to  (a)  replicate  and 
expand  upon  findings  of  DeChillo  et  al.  (1994)  that  suggest 
a relationship  of  parent-professional  collaboration  to 
parents'  satisfaction  with  their  children's  mental  health 
services;  (b)  explore  the  relationship  of  parent-professional 
congruence  on  ratings  of  level  of  functioning  (LOF)  to 
degree  of  collaboration;  and  (c)  examine  the  relationship  of 
parent-professional  congruence  on  ratings  of  LOF  to  parent 
satisfaction  with  services. 


107 


Method 


Participants 

Parents  (N  = 76)  of  youths  receiving  mental  health  services 
at  six  rural  and  urban  community  mental  health  centers 
(CMHCs)  in  Washington  state  participated  in  this  study  as 
part  of  a longitudinal  study  on  regulatory  reform.  Mental 
health  professionals  completed  surveys  on  a total  of  41 
youths.  The  sample  of  youths  was  71%  male,  8%  ethnic 
minority,  and  ranged  in  age  from  4 to  1 8 years  old.  The 
most  common  DSM-IV  diagnoses  were  adjustment  disorder 
(44%),  attention  deficit  disorder  (22%),  and  oppositional 
defiant  disorder  (14%).  Ninety-three  percent  of  the  sample 
was  Medicaid-eligible. 

Instruments 

Parent  Survey 

The  Client  Satisfaction  Questionnaire  (CSQ-8;  Attkinson  & 
Zwick,  1982),  an  8-item  parent  report  of  service  satisfaction, 
originally  developed  by  Hargreaves  and  Attkinson  in  1978, 
is  rated  on  a 4-point  scale.  This  widely  used  scale  has 
demonstrated  adequate  internal  consistency  (alpha  = .93). 

The  PSU  Family/Professional  Collaboration  Scale,  a 
20-item  questionnaire,  was  constructed  by  DeChillo  et  al. 
(1994).  This  scale  asks  parents  to  rate  their  child's  mental 
health  professional's  responsiveness  toward  them  on  a 
4-point  scale.  There  are  five  factors  which  are  considered  in 
this  scale  (i.e.,  supportive  understanding,  accessing  services, 
sharing  information,  utilizing  feedback,  and  overall 
collaboration). 

Parent  and  Professional  Survey 

The  Ecology  Rating  Scale  (ERS)  was  developed  by  quality 
assurance  staff  at  a local  CMHC  with  input  from  family 
members  of  consumers  and  mental  health  providers.  The 
7-item  questionnaire,  rated  on  a 5-point  scale,  measures 
behavioral/emotional  impairments  in  the  following  life 
domains:  (a)  family,  (b)  school,  (c)  emotional,  (d)  legal,  (e) 
recreational,  (f)  health,  and  (g)  social.  This  scale  has 
demonstrated  adequate  internal  consistency  (Srebnik,  1996; 
alpha  = .70  for  parent  raters;  alpha  = .69  for  professional 
raters). 

Results 

Examining  the  relationship  of  parent-professional 
collaboration  to  parental  satisfaction  with  their  children's 
mental  health  services,  a significant  correlation  was  found 
between  the  CSQ  total  score  and  the  total  score  on  the 
Family/Professional  Collaboration  Scale  (r  = .76,  p < .001; 
see  Table  IT 

The  relationship  of  parent-professional  congruence  on 
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The  relationship  of  parent-professional  congruence  on 
ratings  of  LOF  to  degree  of  collaboration  was  also  explored. 
As  a first  step,  differences  between  parent  and  professional 
ratings  of  LOF,  using  the  ERS,  were  calculated  using  t-test 
comparisons.  Parent  and  professional  ratings  were 
significantly  different  in  the  following  domains:  (a)  family, 
(b)  legal,  (c)  health,  and  (d)  social  (see  Table  2). 

The  total  ERS  discrepancy  score  was  then  calculated  (i.e., 
the  difference  between  parent  and  professional  ratings  in 
each  life  domain  of  the  Ecology  Rating  Scale).  This 
discrepancy  score  was  not  significantly  correlated  with  the 
total  score  on  the  Family/Professional  Collaboration  Scale, 

(r  = -.  1 54,  p < ns;  see  Table  \). 

The  relationship  of  parent-professional  congruence  on 
ratings  of  child  LOF  to  parent  satisfaction  with  services  was 
also  explored.  However,  there  was  no  significant  correlation 
found  between  the  total  ERS  discrepancy  score  and  the  CSQ 
total  score  (r  = .161,  p < ns;  see  Table  If 

Discussion 

The  results  of  this  study  reaffirm  the  findings  of  DeChillo  et 
al.  (1994)  that  parent  satisfaction  seems  to  be  related  to 
parent-professional  collaboration.  Parents  and  professionals 
in  this  study,  however,  seemed  to  differ  in  their  perceptions 
of  children's  level  of  functioning  in  several  areas. 
Professionals  perceived  family  and  social  domains  as  areas 
of  greater  child  impairment;  parents  perceived  legal  and 
health  domains  as  areas  of  greater  child  impairment.  One 
explanation  for  these  differences  may  be  professional 
emphasis&endash;given  training  in  family  systems 
approaches  and  negative  peer  group  influences&endash;on 
family  and  social  impairments.  Parents,  embedded  in  the 
family,  may  also  be  less  likely  to  perceive  the  family 
domain  as  a child's  main  problem  area.  Both  of  these 
divergent  perspectives  on  child  functioning  may  be  valid; 
parents  and  professional  should  continue  to  form 
partnerships  where  the  open  exchange  of  ideas  and 
information  is  encouraged. 

Parent-professional  collaboration  does  not  seem  to  be 
associated  with  congruence  of  parent-professional  LOF 
ratings.  One  explanation  for  this  finding  may  be  that  parents 
perceive  the  interpersonal  and  supportive  aspects  of  the 
collaborative  relationship  as  more  important  than  a shared 
view  of  their  child's  functioning.  This  interpretation  is 
supported  in  part  by  the  relationship  found  between  parent 
satisfaction  with  services  and  perceptions  of  collaboration. 

Satisfaction  does  not  seem  to  be  related  to  congruence  of 
parent-professional  LOF  ratings.  Again,  parent  satisfaction 
may  depend  on  a number  of  factors  (e.g.  service  availability 
and  treatment  outcomes),  and  divergent  perspectives  on 
child  LOF  may  not  play  a major  role  in  parent  evaluation  of 
services.  Furthermore,  a global  satisfaction  rating,  such  as 
the  CSQ,  may  be  less  useful  than  a measure  of  satisfaction 
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the  CSQ,  may  be  less  useful  than  a measure  of  satisfaction 
with  individual  service  providers  to  investigate 
parent-professional  agreement.  It  could  also  be  that  the 
relationship  of  satisfaction  to  congruence  in  LOF  ratings  is 
mediated  by  the  degree  to  which  services  are  actually 
provided  to  meet  identified  areas  of  functioning.  A next  step 
to  further  this  line  of  inquiry  would  be  to  test  the 
relationships  of  LOF  rating  congruence,  the  extent  to  which 
services  were  provided  to  meet  identified  need  and  service  ' 
satisfaction.  * 

Overall,  this  study  suggest  that  global  service  satisfaction  is 
more  related  to  characteristics  of  the  parent-professional 
relationship  (i.e.  collaboration)  than  the  degree  of  agreement 
in  how  they  view  the  severity  of  the  child's  difficulties. 
However,  in  order  to  provide  specific  and  appropriate 
services,  the  differences  in  the  "lenses"  through  which 
parents  and  professionals  view  a child's  problems  should  be 
addressed. 
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Introduction 


Within  the  field  of  mental  health,  there  is  an  increasing 
sentiment  that  services  for  families  should  be  designed  to 
promote  empowerment.  Empowerment  has  been 
conceptualized  as  both  a temporary  state  of  being  and  an 
ongoing  process  involving  change  within  the  individual. 
Although  no  single  definition  of  empowerment  has  been 
proposed  that  captures  both  conceptualizations,  a growing 
consensus  has  come  to  see  it  as  a process  through  which 
individuals  gain  control  of  their  lives  through  exerting 
influence  over  their  interpersonal  and  social  environments. 
Within  the  context  of  human  service  delivery  systems,  the 
process  of  empowerment  is  thought  to  occur  when  families 
are  provided  with  opportimities  to  access  the  knowledge, 
skills,  and  resources  that  foster  control  over  their  lives  and 
improve  its  quality  (Singh  et  al.l995). 

A current  model  specifically  designed  to  describe 
empowerment  for  families  who  have  children  with  serious 
emotional  disabilities  was  introduced  by  Koren,  DeChillo, 
and  Friesen  (1992).  This  conceptual  framework  proposed 
two  dimensions  for  empowerment&endash;level  and 
expression.  The  levels  of  empowerment  were  further 
defined  as  Family,  Service  System,  and 
Community/Political.  The  second  dimension,  expression  of 
empowerment,  consists  of  Attitudes  (what  the  parents  feel 
and  believe);  Knowledge  (what  they  know  and  have  the 
ability  to  do);  and  Behavior  (the  actions  of  parents). 

In  light  of  this  model,  Koren,  DeChillo,  and  Friesen  (1992) 
suggested  that  empowerment  might  be  a developmental 
process  where  parents'  focus  moves  from  immediate  family 
concerns,  to  securing  information  and  services  they  need  to 
address  their  child's  need,  and  finally  to  action  to  assist 
other  families  and  address  the  needs  of  all  children.  In  order 
to  examine  this  model,  they  created  the  Family 
Empowerment  Scale  (FES)  which  is  a measure  of 
empowerment  within  families  with  children  having  serious 
emotional  disabilities.  Initially  tested  with  a sample  of  440 
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parents,  factor  analysis  of  the  FES  results  supported  their 
empowerment  model.  To  further  test  this  conceptual 
framework,  Singh,  Curtis,  Ellis,  Nicholson,  Villani,  and 
Weschler  conducted  a study  and  factor  analysis  resulting  in 
a factor  structure  similar  to  the  FES  (1995). 

The  FES  was  selected  as  a measure  of  family  empowerment 
for  the  evaluation  of  the  Access  Vermont  Project,  a 
multilevel  crisis  intervention  jointly  funded  by  a 
Community  Mental  Health  Services  grant  and  Vermont's 
Family  Preservation  Initiative.  The  FES,  however,  was 
previously  tested  with  populations  whose  demographics 
were  quite  different  from  those  of  families  served  by  the 
Family  Preservation  Initiative;  the  current  study  population 
was  poorer,  comprised  of  more  single  parents,  and  fewer 
parents  who  participated  in  support  groups.  This  summary 
describes  assessment  of  the  FES  for  use  with  a population 
more  typical  of  crisis  services-oriented  family  preservation 
programs,  and  includes  comparison  of  our  evaluation's 
factor  results  to  the  factor  structures  and  conceptual 
framework  derived  by  Koren,  DeChillo,  and  Friesen  ( 1 992). 

Method 

Subjects 

Evaluation  participants  included  the  primary  caregivers  of 
the  first  100  eligible  children  and  adolescents  to  complete 
the  intake  procedures  during  the  evaluation  period  of  the 
Access  Vermont  Project.  To  be  eligible  for  inclusion  in  the 
evaluation,  the  child  or  adolescent  had  to  be  referred  for 
crisis  services,  and  determined  to  require  a treatment  team 
and  services  from  two  or  more  community  agencies. 

As  stated  before,  the  demographics  of  these  families 
differed  from  those  participating  in  previous  studies 
utilizing  the  FES.  For  example,  in  the  Koren  et  al.  (1992) 
study,  21%  of  the  families  had  annual  incomes  over  $50,000 
and  only  14%  earned  less  that  $10,000  a year.  Similarly,  in 
Singh  et  al.  (1995),  19%  of  families  had  incomes  over 
$50,000,  with  17%  reporting  incomes  of  $10,000  or  less.  In 
contrast,  86%  of  participants  in  this  study  reported  annual 
incomes  of  less  that  $25,000;  no  families  reported  incomes 
of  over  $50,  000.  Consequently,  these  families  were  more 
likely  to  turn  to  the  public  service  system  for  help  rather 
than  private  therapists.  Additionally,  in  both  Koren  and 
Singh,  many  families  reported  that  they  were  members  of 
support  groups  or  organizations  for  families  with  children 
with  serious  emotional  disabilities.  The  families  in  the 
Access  Vermont  sample  came  to  the  program  in  crisis,  and 
none  reported  participating  in  a support  group  prior  to 
receiving  these  services. 

The  previous  studies  had  a male  population  of  over  70%, 
whereas  males  in  the  current  study  represent  49%  of  the 
sample.  Another  significant  difference  is  in  the  percentage 
of  children  living  in  single  parent  homes.  In  Koren,  et  al. 
(1992),  only  28%  of  the  children  lived  in  single  parent 
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(1992),  only  28%  of  the  children  lived  in  single  parent 
families;  in  our  study,  the  percentage  was  43%.  The 
strongest  similarity  between  our  population  and  those 
participating  in  previous  studies  was  in  race;  over  75%  of 
the  subjects  of  all  three  studies  discussed  here  were 
Caucasian. 

Clinically,  we  found  that  over  50%  of  the  children 
participating  in  the  evaluation  of  the  Family  Preservation 
Initiative  scored  in  the  clinical  range  for  aggressive  and 
delinquent  behaviors.  Further,  43%  of  the  children  exhibited 
attention  problems  and  36%  had  problems  related  to  thought 
disorders. 

After  receiving  informed  consent  from  the  participants, 
demographic  information,  a Child  Behavior  Checklist 
(Achenbach  & Edelbrock,  1991),  and  other  measures  were 
collected  by  the  intake  worker  from  the  child  or  adolescent's 
primary  caregiver.  Within  two  weeks  of  the  intake  an 
evaluation  team  member  telephoned  the  primary  caregiver 
and  collected  information  which  included  the  FES  and 
Family  Satisfaction  Questionnaire. 

The  FES  consists  of  34  items  rated  on  a 5-point  Likert-type 
scale  from  1 = not  true  at  all  to  5 = very  true,  designed  to 
measure  two  dimensions  of  a family's  empowerment, 
empowerment  related  to  various  system  levels  and  the 
manner  in  which  a family  expresses  empowerment. 

Results  and  Discussion 

Though  the  demographic  characteristics  of  our  population 
were  very  different  from  the  previous  populations  examined 
(Koren,  DeChillo,  & Friesen,  1992;  Singh  et  al.,  1995),  the 
factors  derived  were  very  similar;  three  factors  were  derived 
which  were  almost  identical  to  the  four  factors  derived  by 
Singh  et  al.  (1995;  see  Table  1).  The  major  difference  was 
that  the  present  solution  has  three  factors  while  their 
solution  had  four  factors,  and  the  items  in  the  factor  they 
labeled  knowledge  were  distributed  over  the  self-efficacy, 
system  advocacy,  and  competence  factors  for  our  results. 
Since  only  one  item  (i.e.,  other  than  those  contained  in  the 
knowledge  factor)  moved  to  another  factor,  we  used  the 
same  factor  labels  as  Singh  et  al.  (1995).  Factor  1, 
self-efficacy,  reflects  the  primary  caregiver's  perception  of 
her/his  ability  to  obtain  needed  services  from  the  children's 
mental  health  system  for  her/his  child.  Factor  2,  system 
advocacy,  represents  the  primary  caregiver's  opinion  on  how 
effectively  she/he  can  be  an  agent  for  change  in  the 
children's  mental  health  system,  and  Factor  3,  Competence, 
refers  to  the  primary  caregiver's  feelings  of  competence  as  a 
parent.  The  items  that  moved  from  the  knowledge  factor 
seem  to  have  moved  to  related  factors  (e.g.,  the  items  that 
relate  to  knowledge  about  system  advocacy  seem  to  have 
moved  to  the  system  advocacy  factor). 

Based  on  our  results,  the  three  factor  solution  was  the  most 
suitable,  because  it  was  most  conceptually  meaningful  and 
the  statistical  properties  were  sound.  The  solution  explained 
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the  statistical  properties  were  sound.  The  solution  explained 
42%  of  the  total  variance  and  the  alpha  coefficients  for  the 
factors  indicate  substantial  internal  consistency.  However, 
the  Pearson  product-moment  correlations  among  the  three 
factors  are  moderately  high  with  all  the  correlations  being 
significant  at  the  p < .001  level.  This  moderately  high 
correlation  of  all  the  derived  factors  indicates  that  the 
factors  are  not  independent  (see  Figure  1). 

The  FES  has  been  characterized  as  being  a useful  tool  for 
longitudinally  evaluating  programs  intended  to  assist  family 
development  related  to  the  acquisition  of  knowledge,  skills, 
services,  and  resources  from  the  mental  health  system  for 
children  (Singh  et  al.,  1995).  We  view  this  assertion  very 
cautiously  for  two  of  the  derived  factors.  An  examination  of 
Figure  2 demonstrates  that  the  self-efficacy  and  competence 
factors  are  highly  skewed  to  the  upper  end  of  the  scale  for 
this  baseline  measure.  This  indicates  for  our  evaluation  that 
it  will  be  extremely  difficult  to  measure  increases  in  these 
dimensions  of  empowerment  should  they  occur;  also,  a 
regression  to  the  mean  may  even  be  expected.  The  system 
advocacy  factor,  however,  is  more  evenly  distributed,  and  it 
may  be  much  more  useful  for  measuring  change  over  time. 

Finally,  we  examined  the  relationship  of  perceived  family 
empowerment  to  various  demographic,  behavioral, 
satisfaction,  and  risk  variables  (see  Figure  31.  Significant 
differences  were  found  between  the  highest  scoring  and 
lowest  scoring  primary  caregivers  on  the  FES  for  three 
variables.  Belief  in  the  proposition  that  the  family  and  child 
will  be  unconditionally  supported  in  services  and 
satisfaction  with  services  were  related  to  higher  perceived 
self-efficacy  and  ability  to  advocate  for  improved  children's 
mental  health  services.  High  levels  of  perceived  child 
behavior  problems  were  highly  associated  with  feelings  of 
lack  of  competence  in  child  rearing.  If  we  assume  that  these 
associated  variables  are  situational  and  subject  to  change  in 
either  a positive  or  negative  direction,  then  it  is  also  possible 
that  the  aspects  of  family  empowerment  measured  by  the 
FES  are  also  situational  and  not  developmental.  We  will  be 
examining  this  possibility  as  we  track  this  evaluation  group 
over  the  next  two  years. 

The  FES,  as  with  all  the  measures  described  in  this  study, 
measure  the  primary  caregiver's  perspective.  There  have 
been  no  independent  observations  made.  However,  in 
discussions  with  therapists  who  provide  direct  services,  they 
have  noted  that  their  impressions  of  the  empowerment  of 
the  primary  caregiver,  especially  as  it  relates  to  competence 
as  a parent,  may  be  very  different  from  the  perspective  of 
the  prim^  caregiver.  This  is  an  indication  that  independent 
observation  of  primary  caregivers  will  be  necessary  to 
measure  their  levels  of  empowerment  and  changes  in  actual 
behavior. 

The  data  presented  here  is  from  the  interviews  that  were 
completed  within  two  weeks  after  intake  and  are  based  on 
only  the  first  100  families  evaluated.  Therefore,  the  results 
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are  preliminary  and  the  data  will  be  reanalyzed  when  there 
is  a larger  sample  to  draw  from  which  will  provide  a more 
stable  factor  solution.  These  families  will  be  interviewed 
again  at  6 months,  one  year,  and  two  years  from  the  time  of 
intake  to  determine  if  they  are  experiencing  any 
developmental  changes  in  their  reports  of  empowerment. 
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Introduction 


The  prevalence  of  youth  demonstrating  serious  behavior 
problems  continues  to  grow  at  an  alarming  rate,  creating  a 
tremendous  challenge  for  our  schools  and  communities 
(Sixteenth  Annual  Report  to  Congress,  1994).  Given  the 
dramatic  cuts  in  human  services  for  children  and 
adolescents,  there  is  a growing  need  to  identify  alternative 
models  for  delivering  effective  and  accountable  services  for 
youth  and  families  with  fewer  resources. 

The  Community  Collaboration  Project  represents  an  effort . 
to  maximize  resources  within  one  community  in  rural 
Vermont  by  creating  an  interagency  consultation  model 
designed  to  maintain  students  with  SED  in  their  local 
schools  and  communities.  The  principles  of  "Wraparound" 
were  applied  to  create  community  based  consultation 
practices  and  interventions  that  were  interagency  focused, 
family-centered,  individualized,  strength  based, 
preventative,  and  reflective  of  multiple  life  domains 
(Burchard  & Clarke,  1990).  The  model  emphasized 
consultation  instead  of  direct  service  in  order  to  serve  a 
greater  number  of  children  and  families  and  create  broader 
systems  change. 

Model  Overview 

The  foundation  for  this  project  was  an  Interagency  Support 
Team  (1ST)  which  provided  consultation  and  training  to 
community  teams,  focusing  on  both  the  individual  student 
and  broader  systemic  needs.  Project  funds  were  used  to  gain 
release  time  for  four  members  of  the  community:  (a)  family 
consultant  &endash;  a parent  of  a child  who  has  experienced 
a serious  emotional  disturbance;  (b)  education  consultant 
&endash;  an  educator  representing  the  three  supervisory 
unions  in  the  county;  (c)  mental  health  consultant  &endash; 
a clinician  associated  with  the  local  community  mental 
health  center;  and  (d)  social  services  consultant  &endash;  a 
representative  of  the  district's  social  services  department. 
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For  twenty  hours  per  week,  these  four  individuals  were 
released  from  their  current  responsibilities  to  serve  as  the 
region's  1ST.  The  1ST  met  regularly  with  the  county's 
directors  of  mental  health,  social  services  and  education  to 
review  patterns  of  need  and  identify  policies  and  practices 
that  posed  barriers  to  effective  inclusion  of  identified 
children  and  youth.  To  foster  systems  change  on  multiple 
levels,  the  1ST  spent  its  first  year  designing  and  conducting 
a community  needs  assessment  and  action  planning  process. 
Based  on  the  information  from  this  process,  numerous 
workshops  (e.g.,  collaborative  teaming,  conflict  resolution, 
developing  interagency  coordinated  service  plans,  and 
family  advocacy)  were  offered  on  a regular  basis  throughout 
the  county.  In  addition,  the  project  was  involved  in  the 
reorganization  of  community  governance  to  better  serve 
children  and  families.  Concurrently,  the  1ST  provided 
consultation  to  community-based  teams  on  issues 
concerning  individual  students  and  broader  systems  level 
concerns.  The  primary  focus  of  this  summary  will  be  the 
consultation  provided  to  community  teams. 

Methods 

Participants 

This  project  was  implemented  in  Addison  County,  Vermont, 
a rural  region  in  central  Vermont,  with  a population  of 
approximately  33,000.  Three  school  districts,  composed  of 
5161  students  (grades  K-12),  received  services.  At  the  start 
of  the  project,  67  children  and  adolescents  in  Addison 
County  were  in  out-of-home  placements. 

Referrals  for  consultation  around  both  individual  students 
and  broader  systems  issues  were  accepted.  For  individual 
student  referrals,  the  major  criteria  for  acceptance  included: 
(a)  the  student  must  be  identified  as  having  an  emotional 
and/or  behavioral  disorder  (as  defined  by  either  the  special 
education  definition  or  ACT  264  legislation,  1988);  (b)  the 
student  must  be  at  high  risk  of  removal  from  his  or  her 
home,  school,  or  community;  and  (c)  there  must  be  a 
willingness  on  the  part  of  those  involved  to  work  together  as 
a team,  with  the  objective  of  trying  to  maintain  the  student 
within  the  community.  With  respect  to  system  referrals,  the 
criteria  included:  (a)  the  system's  issues  must  impact  on 
students  with  SED,  and  (b)  there  must  be  a willingness  to 
work  in  a collaborative  teaming  fashion. 

A total  of  16  referrals  were  accepted  involving  12  individual 
students  and  4 broader  systems  issues.  Individual  referrals 
involved  students  in  grades  K-12.  The  families  of  these 
students  reflected  predominantly  low  socio-economic 
backgrounds  (i.e.,  relatively  low  incomes,  educational,  and 
occupational  status);  over  one-third  were  single  parent 
homes,  and  one-third  of  the  students  were  involved  with 
social  services.  Of  the  systems'  referrals,  two  came  from 
special  education  staff,  one  from  an  early  education 
program,  and  one  from  the  staff  at  the  local  community 
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program,  and  one  from  the  staff  at  the  local  community 
mental  health  center. 

Project  Interventions 

Two  project  staff  were  assigned  to  each  referral.  For 
individual  student  referrals,  the  initial  step  included  the 
formation  of  a plaiming  team  whose  composition  minimally 
included  the  student,  the  parents,  educators,  and  service 
providers  from  relevant  agencies.  Weekly  team  meetings 
were  encouraged  to  reinforce  a preventative  focus. 

Emphasis  was  placed  on  ensuring  that  parents  were  equal 
members  of  the  teaming  process.  The  second  stage  involved 
a comprehensive  ecological  assessment  to  gather 
background  and  current  information  reflecting  all  aspects  of 
the  student's  life.  The  team  focused  on  determining  the 
student's  strengths,  as  well  as  areas  of  need.  Based  on  this 
information,  a comprehensive  plan  was  developed  that  built 
on  these  areas  of  strength,  and  addressed  the  needs  of  the 
student,  the  family,  and  the  service  providers.  Resources 
were  pooled  from  all  of  the  involved  agencies.  Plans  were 
monitored  on  an  ongoing  basis  so  that  modifications  could 
be  made  as  necessary.  The  length  of  intervention  varied 
according  to  the  needs  of  the  student  and  team. 

System  referrals  also  made  use  of  a collaborative  teaming 
model,  with  all  key  persons  involved  and  meetings  held  on  a 
regular  basis.  Similar  to  the  process  for  individual  referrals, 
an  initial  comprehensive  assessment  was  completed  to 
determine  both  the  strengths  and  needs  of  the  team  members 
and  the  system  aspects  involved.  Duration  of  intervention 
was  dependent  on  need. 

Evaluation  Measures 

Both  quantitative  and  qualitative  evaluation  approaches 
were  used  to  assess  the  efficacy  of  this  model.  With  respect 
to  the  quantitative  evaluation,  all  members  of  the  team  were 
given  instruments  to  complete  on  a pre-post  basis.  The 
Consultation  Satisfaction  Questionnaire  was  used  as  a post 
assessment.  All  of  the  measures  listed  below  were 
administered  for  individual  student  referrals.  The  Teaming 
Satisfaction  and  Consultation  Satisfaction  Questionnaires 
were  given  for  systems  referrals. 

• Demographic  Survey:  assesses  socio-economic 
indicators  with  respect  to  the  student's  family; 
completed  by  the  parent(s). 

• Child  Behavior  Checklist  (CBCL;  Achenbach,  1991), 
completed  by  the  parent(s);  Teacher  Report  Form 
(TRF;  Achenbach,  1991),  completed  by  the 
teacher(s);  and  Youth  Self  Report  (YSR;  Achenbach, 
1991),  completed  by  the  youth  if  age  1 1 years  or 
older:  behavioral  checklists  used  to  obtain  a global 
assessment  of  the  student's  emotional  and  social 
functioning. 

• Family  Support  Scale  (Dunst,  Trivette,  & Deal, 

1988):  a measure  of  perceived  family  support 
completed  by  the  family. 
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completed  by  the  family. 

• Academic  records:  attendance,  suspensions,  academic 
performance,  and  achievement  scores. 

• Parent  Satisfaction  Survey  and  Youth  Satisfaction 
Survey:  both  designed  specifically  for  this  project  to 
assesses  satisfaction  with  all  services  received; 
completed  by  parent(s)  and  youth  respectively. 

• Teaming  Questionnaire:  designed  specifically  for  this 
project  to  assess  satisfaction  with  the  teaming  process 
and  interagency  collaboration;  completed  by  all  team 
members. 

• Consultation  Satisfaction  Questionnaire:  designed 
specifically  for  this  project  to  assess  satisfaction  with 
the  services  of  project  staff;  completed  by  all  team 
members. 

The  qualitative  component  of  the  evaluation  consisted  of 
semi-structured  interviews  with  all  team  members  for  three 
select  students  to  explore  systems  issues  in  a more  in-depth 
fashion. 

Results 

Quantitative 

According  to  the  data  from  the  CBCL,  TRF,  and  YSR, 
significant  pre-  and  post-  differences  (p  <.01)  were  noted 
only  with  respect  to  teacher  perceptions  tsee  Table  IT  In 
particular,  significant  decreases  were  reported  on  scales 
relating  to  externalizing  problems  (e.g.,  aggressive  behavior 
and/or  delinquency).  No  significant  change  scores  were 
noted  for  the  Internalizing  scales  on  the  TRF,  nor  for  any  of 
the  broad-band  scales  of  the  CBCL  or  YSR.  Data  on 
academic  performance  is  yet  to  be  analyzed. 

Based  on  the  data  from  the  Family  Support  Scale,  there  was 
a trend  towards  increased  social  support  from  both  informal 
(i.e.,  extended  family  and/or  friends)  and  professional 
resources.  While  this  change  score  did  not  meet  the 
traditional  p = .05  cut-off,  the  p score  of  .10  is  significant 
for  this  small  sample  size.  Significant  increases  at  p < .05 
were  also  found  with  respect  to  almost  all  individual  items 
on  the  Parent  Satisfaction  measure  (see  Table  2).  Regarding 
teaming,  parents  felt  more  listened  to,  more  involved  in 
decision-making,  more  respected  by  other  team  members, 
experienced  more  equality  as  a team  member,  attended  more 
meetings,  and  were  satisfied  with  their  child's  progress. 
Increases  in  overall  satisfaction  and  satisfaction  with  their 
family  situation  were  noted  at  the  p < .10  level.  On  the 
Youth  Satisfaction  Measure,  the  only  significant  item  was 
an  increase  in  perceived  choice  of  services  (p  <.05).  Overall, 
there  was  a high  degree  of  satisfaction  with  project 
involvement  from  youth,  parents,  educators,  and  other 
service  providers.  On  a 5 point  Likert-type  scale  of 
Consultation  Satisfaction,  the  mean  group  scores  were  as 
follows:  (a)  parents,  4.1;  (b)  youth,  5.0;  (c)  regular 
educators,  4.4;  and  (d)  special  educators,  4.3. 
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In  terms  of  educational  placement,  at  the  time  of  referral,  10 
students  were  served  in  their  regular  public  school 
mainstream  program,  1 was  home-schooled,  and  1 was 
served  in  an  alternative  education  program  within  the 
regular  public  school.  At  the  end  of  the  project,  10  students 
were  served  within  the  regular  public  school,  one  was  in  an 
alternative  education  program,  and  one  student  was  placed 
in  a residential  school. 

Qualitative 

The  qualitative  interviews  complemented  the  above  results. 
Consistent  pro-active  team  meetings  were  viewed  as  a 
critical  component  in  the  success  of  a student's  planning 
effort.  Parents  emphasized  the  importance  of  being  viewed 
as  equal  team  members,  with  equal  decision-making  power. 
The  most  successful  efforts  gave  considerable  support  to  the 
direct  care  providers  (i.e.,  families,  teachers,  individual 
assistants,  mental  health  workers,  and  social  workers). 
Interagency  composition  of  the  team,  willingness  to  share 
resources,  and  flexibility  in  planning  were  also  associated 
with  a greater  likelihood  of  student  success.  The  nature  of 
the  student's  behavioral  disability  did  not  appear  to  be 
predictive  of  outcome. 

Discussion 

Based  on  these  preliminary  results,  it  appears  that  the 
interagency  consultation  model  has  the  potential  for  being 
an  effective  use  of  resources  for  supporting  children  and 
adolescents  with  severe  emotional  disturbance  within  their 
local  schools  and  communities.  In  this  time  of  limited 
funding  for  education  and  human  services,  we  can  no  longer 
afford  to  operate  as  independent  agencies,  each  developing 
and  implementing  separate  plans  for  youth  and  families. 
Through  this  model,  agencies  and  families  are  encouraged 
to  work  together,  building  the  capacity  of  direct  line 
workers  and  maximizing  the  resources  within  a community. 
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This  study  was  designed  to  document  the  processes  and 
dynamics  associated  with  the  development  and  functioning 
of  two  multidisciplinary  teams  created  in  response  to  the 
Access  Vermont  Initiative. 

Interagency  and  multidisciplinary  teams  are  not  new 
entities.  During  the  past  few  years,  researchers,  health  care 
professionals,  educators,  and  business  people  as  well  as 
others,  have  voluntarily  and/or  been  mandated  to  form 
working  teams.  The  logic  supporting  the  formation  of  teams 
is  that,  theoretically,  a group  of  people  pooling  knowledge, 
experience,  resources,  and  energies  should  be  more 
productive,  produce  a more  comprehensive  product,  and  be 
more  cost  efficient.  In  short,  a positive  and  synergistic  effect 
is  expected. 

Much  has  been  written  regarding  the  hypothesized  benefits 
and  efficacy  of  the  collaborative  and  multidisciplinaiy  team 
and  "system  of  care"  approach  to  the  treatment  of  children 
with  severe  emotional  disturbances.  However,  there  is  a 
paucity  of  research  related  to  the  processes  and  dynamics 
associated  with  interagency  team  development  and  growth. 

O'Looney  (1995)  distinguishes  between  the  two  terms 
"collaboration"  and  "service  integration"  that  are  often  used 
interchangeably  in  literature.  O'Looney  defines 
collaboration  as  the  processes  or  dynamic  associated  with  a 
system  and  which  serve  to  promote  integration;  whereas 
service  integration  is  defined  as  the  system  itself  Existing 
literature  has  tended  to  focus  on  the  system  rather  than  on 
the  process.  The  processes  related  to  collaboration  include 
shared  goals  and  leadership,  the  development  of  negotiation 
skills,  consultation,  conflict  resolution  skills,  egalitarianism. 


interpersonal  relations,  and  a respect  for,  value  of,  and 
empowerment  of  all  team  members.  O'Looney  (1995) 
further  suggests  that  a successful  collaboration  will  involve 
a relinquishing  by  individual  members  of  a measure  of 
autonomy  and  the  development  of  a more  global  and  less 
personalized  agenda  or  perspective;  a process  that  might 
prove  threatening  for  some  team  members. 

The  current  researchers  suggested  that  a historical  and 
almost  biographical  account  of  the  developmental 
experience  of  the  two  Access  Vermont  teams  would  identify 
a pattern  of  growth,  a dynamic  unique  to  each  team,  as  well 
as  elements  shared  in  a team  experience.  This  study, 
therefore,  focused  on  the  process  rather  than  on  the  system. 

Background 

In  January,  1994,  the  State  Interagency  Team  invited  Local 
Interagency  Teams  to  participate  in  an  initiative  designed  to 
"promote  family  preservation."  Local  team  coordinators 
were  asked  to  convene  a meeting  of  families  and  regional 
service  providers  to  develop  a plan  for  promoting  family 
preservation  including  a supporting  management  structure 
(a  Stakeholders'  Team),  and  submit  this  resulting  proposal 
in  application  for  financial  assistance  (Access  Vermont 
funding).  From  this  initial  planning  meeting,  regional  or 
stakeholders'  teams  would  evolve  whose  responsibility  it 
became  to  oversee  systems  and  financial  planning,  program 
and  human  resource  development,  and  also  to  monitor  and 
evaluate  outcomes  resulting  from  the  implementation  of  the 
community  plan.  Twelve  catchment  areas  throughout 
Vermont  were  awarded  Access  Vermont  initiative  funding. 

Part  of  the  expectation  at  the  state  level  was  that  various 
service  organizations  involved  in  serving  children  with 
serious  emotional  disabilities  and  their  families  would 
collaborate  in  order  to  provide  a more  comprehensive  and 
efficient  mental  health  service  delivery  system.  This 
increased  efficiency  would  be  facilitated  by  providing  a 
multi-access  point  of  entry  to  services.  The  initially 
contacted  agency  or  service  provider  would  offer  to  contact 
the  appropriate  agency  or  agencies  for  the  family  and  thus 
provide  a point  of  access  to  multiple  services. 

The  plans  submitted  by  both  of  the  two  community  teams 
described  in  this  study  included  the  stated  objectives  of: 

• A reduction  in  numbers  of  children  placed  in  state 
custody; 

• the  promotion  of  family  involvement  in  the 
development  of  plans  for  services;  and 

• the  development  of  a referral  process  involving  a 
multi-access  point  of  entry. 

Both  of  these  catchment  areas  serve  quite  large  geographical 
areas.  Both  regions  experience  relatively  high  rates  of 
unemployment,  child  poverty,  teen  pregnancy,  and  child 
abuse  compared  with  other  Vermont  regions.  Working 
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members  on  the  Stakeholders'  Team  in  both  catchment  areas 
included  a member  from  Mental  Health  Services,  Youth 
Services,  Drug  and  Alcohol  and  Social  Rehabilitation 
Services  (SRS).  Additional  members  on  Team  II  included 
delegates  from  the  Police,  Education,  the  Department  of 
Health,  Domestic  Violence  and  one  active  parent 
participant.  Team  I included  only  one  additional  member,  a 
local  hospital  administrator.  Two  working  team  members 
served  as  representatives  on  both  Team  I and  Team  II. 
Interviews  were  also  conducted  with  two  former  members 
of  Team  I,  including  a member  of  the  local  Police  force  and 
a parent  representative.  Both  Stakeholders'  Teams  have  been 
meeting  for  less  than  a year,  but  many  members  of  these 
smaller  teams  were  involved  as  members  in  the  larger 
Governance  or  Access  Planning  Teams. 

Method 

The  researchers  investigated  the  experience  and  perceptions 
of  Stakeholders'  Team  members  using  a qualitative 
methodology.  The  principle  source  of  data  involved  the  use 
of  a semi-structured  interview  format.  Team  members  were 
interviewed  either  in  person  (6  participants)  or  by  telephone 
( 1 0 participants)  and  were  presented  questions  related  to 
team  involvement  and  experience  as  well  as  their 
perceptions  related  to  degree  of  collaboration  associated 
with  the  team.  Secondary  sources  of  data  included  field 
notes  and  investigator's  observations  resulting  from 
attendance  at  Stakeholders'  Team  meetings,  team  meeting 
minutes,  and  planning  applications.  Participants'  interviews 
were  taped,  transcribed,  and  coded  by  two  independent 
raters,  and  resulting  themes  were  recorded. 

In  this  pilot  study,  three  researchers  conducted  the 
interviews,  collected  field  notes,  and  reviewed  minutes  and 
funding  proposals  (see  Appendix  for  interview  questions). 

Results  and  Discussion 

These  two  Access  Vermont  management  teams  were  in  the 
early  stages  of  development.  This  study,  then,  may  serve  as 
a building  block  for  future  investigation  into  ongoing  team 
growth  and  process.  Members  from  both  teams  reported  that 
team  building  is  a process;  three  members  from  Team  II 
described  this  process  as  evolutionary.  When  asked  whether 
or  not  the  team  was  fulfilling  its  purpose  or  goal,  one 
member  responded,  "I  think  that  question  is  premature  at 
this  point.  The  team  and  its  goals  are  evolving  and  in  a very 
positive  way." 

As  the  transcriptions  were  analyzed,  five  prominent  themes 
emerged  which  spoke  to  the  research  question  regarding  the 
development  of  team  unity  and  collaborative  processes; 

1 . Issues  of  trust. 

2.  Professional  and  personal  networking. 

3.  Unifying  goals  and  a common  vision. 

4.  Communication  and  negotiation  skills. 
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4.  Communication  and  negotiation  skills. 

5.  Inclusive  community  representation. 

1.  Issues  of  Trust 

Trust  is  a transactional  construct  involving  a sense  of  the 
value  of  the  contributions,  experience,  and  knowledge  of 
another  or  others  who  in  turn  reflect  this  confident  reliance. 
Trust  necessarily  springs  from  a shared  sense  of  security  and 
faith  and  that  one's  beliefs  and  priorities  will  not  be 
betrayed.  Without  the  element  of  trust  and  the  positive 
perception  of  interdependence  among  and  between  team 
members,  the  collaborative  processes  will  be  impeded. 
Previous  research  considering  interdisciplinary  collaborative 
efforts  have  identified  issues  of  trust  as  being  paramount  to 
team  development  (Blechert,  Christiansen,  & Kari,  1987; 
Melaville,  Blank,  8c  Asayesh,  1993;  O'Looney,  1995). 
Territorial  or  competitive  attitudes  were  identified  by  both 
teams  as  early  obstacles  inhibiting  feelings  of  trust. 

A voice  from  Team  II  articulated  this  theme: 

"I  think  there's  enough  trust  in  the  group  that  people  have 
been  able  to  put  things  on  the  table  and  deal  with  them  as 
they  come  up." 

A second  voice  from  the  same  team: 

"Most  of  us  that  have  been  on  this  team  had  been  on  other 
teams  together,  so  we've  built  up  our  relationship  from 
there.  I think  we  had  already  established  a level  of  trust 
necessary  for  really  the  type  of  team  that  we  have  now.  So,  I 
think  that  it  was  really  our  own  experience  on  other  teams  in 
the  community  that  really  helped  us  in  our  collaborative 
process.  We're  open  to  listening  to  each  other's  perspectives 
and  willing  to  change  our  own. 

Confidentiality  is  basic  to  the  development  of  trusting 
relationships.  A parent  identified  her  feelings  of  lack  of 
inclusion  as  a member  of  one  team  as  directly  related  to  the 
team's  failure  to  invest  trust: 

"I  was  a bank  manager  «&endash;  confidentiality  is  one  of 
my  better  things.  I'm  on  the  school  board.  I have  an  autistic 
child.  I work  with  special  education.  There's  a lot  of  issues 
related  to  confidentiality  that  I've  lived  through.  If  you  don't 
trust  the  parent,  then  there's  no  sense  in  having  them  on 
your  committee." 

Trust  and  confidence  at  the  state  and  client  levels  were 
addressed.  It  appeared  that  the  consolidation  of  trust  at  the 
team  level  made  increasingly  apparent  the  need  for  the 
development  of  trusting  relationships  at  multiple 
organizational  levels. 

One  voice  addressed  the  need  for  state  administrators  to 
demonstrate  faith  in  the  competence  of  local  teams; 

"I  think  giving  as  much  information  as  is  known.  So  being 
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"I  think  giving  as  much  information  as  is  known.  So  being 
really  forthrightE  not  having  hidden  agendas.  If  you  know 
something's  coming  down  the  pike,  just  tell  us  something's 
coming  down  the  pike.  And  we'll  deal  with  it.  And  also 
respecting  our  ability  to  deal  with  issues." 

Team  members  also  expressed  the  need  to  extend  this 
atmosphere  of  faith  to  the  families  they  serve: 

"I  think  most  people  really  do  feel  that  people  [families] 
need  choices  so  that  people  are  able  to  pick  who's  best  able 
to  meet  their  needs.  And  I think  we  need  to  trust  in  people's 
ability  to  figure  out  what  they  need." 

2.  Professional  and  Personal  Networking 

O'Toole  and  Montjoy  (1984)  operationalize  networking  as 
"the  actions  of  people  working  together  on  specific  tasks 
that  solidify  collaboration  and  build  a sense  of 
interdependence  that  many  who  have  studied  collaboration 
identify  as  a key  ingredient  in  multi-action  implementation" 
(page  #).  Many  informants  identified  networking  and  task 
sharing  as  critical  contributors  to  team  unity. 

"I  think  bringing  people  together  to  discuss  common  issues 
and  concerns  and  offer  opportunities  to  brainstorm  and  to 
work  out  problems  [promotes  team  unity  and  benefits  the 
target  population].  And  I think  there's  other  things  which 
obviously  evolve  from  that  &endash;  support  to  families, 
support  to  the  community,  to  school  personnel,  and 
therefore  it  increases  the  understanding,  the  patience,  and 
the  support  by  having  the  appropriate  people  involved." 

Provision  of  a forum  in  which  to  disclose  and  share  personal 
and  professional  frustrations  prompted  a spirit  of 
networking: 

"Just  getting  things  in  place  and  talking  to  each  other  and 
working  out  the  bumps.  Everybody  has  kind  of  hung  in 
together  and  there  have  been  times  when  we  all  want  to  pull 
our  hair  out  but  people  have  continued  to  just  sit  down  and 
talk  to  each  other  and  put  it  on  the  table  and  say,  'This  is 
what's  making  me  crazy  and  can  we  do  something  about 
this.'  And  we've  seemed  to  be  continually  doing  that  and 
able  to  do  it  and  that's  very  satisfying." 

A second  individual  described  networking  as  pieces  of  a 
puzzle  coming  together  to  create  a more  efficient  whole. 

"I  don't  think  any  one  of  us  provides  what  every  family 
needs.  Even  if  we  had  unlimited  funds,  I don't  think  any  one 
agency,  group,  church,  religion,  or  whatever  group  we  come 
from  can  be  all  to  all  people.  Who  can  offer  what  piece  and 
share  information  with  each  other  so  we  don't  go  off  on 
tangents.  If  we  were  in  our  own  little  worlds,  I think  we'd 
duplicate  more." 

Collaboration  includes  a relinquishing  of  some  portion  of 
personal  or  professional  status  or  position,  a resolution  of 
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personal  or  professional  status  or  position,  a resolution  of 
territorial  issues,  and  an  adoption  of  a team  identity. 
Networking,  an  important  component  of  team  building 
processes,  appears  to  promote  creation  of  a team  identity,  as 
well. 

"I  think  it's  going  out  into  the  community  together.  It's 
developing  that  identification  as  being  Access,  and  you  tend 
to  stop  identifying  yourself  as  SRS  or  Mental  Health  and 
you  start  identifying  yourself  as  Access  and  that's  an 
important  component  in  collaborating  and  team  building 
and  not  only  for  the  team  itself,  but  within  the  community 
and  among  those  agencies." 

3.  Unifying  Goals  and  a Common  Vision 

Research  in  team  development  and  multi-agency 
collaboration  has  emphasized  the  importance  of  shared 
goals  and  a common  vision  (George,  1987;  Mattessich  & 
Monsey,  1993).  The  emergence  of  a shared  mission  or  an 
ideal  is  an  integral  part  of  the  process  of  team  building.  As 
Senge  (1990)  points  out: 

"Visions  that  are  truly  shared  take  time  to  emerge.  They 
grow  as  a by-product  of  interactions  of  individual  visions. 
Experience  suggests  that  visions  that  are  genuinely  shared 
require  ongoing  conversation  where  individuals  not  only 
feel  free  to  express  their  dreams,  but  learn  how  to  listen  to 
others'  dreams.  Out  of  this  listening,  new  insights  into  what 
is  possible  emerge." 

These  wisdoms  were  echoed  in  the  transcribed  experiences 
of  group  members.  . One  group  member  responded: 

"I  think  that's  a place  that  we're  coming  to. . . to  begin  to 
describe  a vision.  I think  that  we're  all  moving  in  the  same 
direction,  but  I can't  really  say  that  &endash;  'Yes, 
absolutely  we  are  and  this  is  our  vision.'  That's  the  kind  of 
developmental  place  that  we  are.  I don't  feel  like  it's 
something  that's  lacking.  I think  that  we've  been  busy  doing 
other  work  and  that  we're  now  at  a point  where  we  can  step 
back  and  say  &endash;  'Okay,  where  are  we  going  ?'  and 
'What's  our  vision?'  I think  that  goes  back  to  my  desire  for 
some  time  for  planning  and  thinking  and  dreaming  about 
what  we  are  going  to  do  and  how  we  are  going  to  be  most 
effective.  Personally,  I would  like  to  try  to  involve  kids  and 
families  and  the  commvmity  together  in  a way  that 
incorporates  kids  more  in  what's  going  on,  so  that  they  feel 
valued,  and  they  feel  wanted  and  feel  they  have  a stake  in 
the  community  and  a stake  in  being  here." 

Although  common  goals  and  a vmifying  vision  have  clearly 
been  identified  as  essential  in  team  development,  they  are 
also  part  of  a growing  process.  Team  members  recognized 
that  a sense  of  mission  is  not  static,  but  rather  it  evolves 
with  maturity.  One  informant  defined  the  vision  as: 

"...  a little  bit  fuzzy  right  now  [but  describes]  a sense  that 
there's  something  bigger  out  there  that  we  need  to  reach. . . 
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there's  something  bigger  out  there  that  we  need  to  reach. . . 
to  go  beyond  and  to  reach  some  of  the  outcomes  that  we're 
talking  about,  because  otherwise  I think  we're  just  going  to 
keep  doing  the  same  thing." 

4.  Communication 

The  fourth  theme,  communication,  has  been  identified  in 
research  as  essential  to  team  cohesion  (Chafetz,  West,  & 
Ebbs,  1988;  Mattessich  & Monsey,  1993).  Communication 
was  defined  as  the  channels  through  which  information  is 
disseminated.  Effective  communication  is  characterized  by 
inclusive  and  respectful  sharing  of  pertinent  information. 

Participants  informed  us  that  it  is  not  only  intrateam 
communication  and  information  flow  that  impacts  on  team 
effectiveness,  but  equally,  the  channels  carrying  information 
between  state  and  local  levels.  Team  members  called  for 
communication  clearly  delineating  and  defining 
expectations  and  boundaries. 

"If  actions  initiated  at  the  state  level  are  just  going  to  be 
countermanded  later  on  then  it's  not  going  to  be  very 
worthwhile  to  put  in  a lot  of  effort  at  the  local  level.  It's  a 
waste  of  time.  So  getting  direction  up  front  and  honest  clear 
guidelines  of  what  is  going  to  happen  is  very  worthwhile. 
'What  are  the  objectives?  What  is  the  point  of  having  the 
team?  What  are  the  parameters  of  our  authority?'  Really 
define  things  and  being  really  clear  about  what  we  are 
addressing." 

Reiterating  this  concern  related  to  state  and  local  lines  of 
communication,  this  participant  used  the  example  of  a 
proposed  announcement  calling  for  redistricting  as 
indicative  of  poor  communication. 

"For  Governance  Boards  not  to  know  about  that,  I think  is 
unconscionable  and  it  says  that  [it  is  assumed]  that  their 
maturity  is  not  at  a level  where  they  can  handle  that 
information.  We  need  to  know  [state  expectations  and 
projections]  right  from  the  start.  What  is  it  that  the  state  will 
give  to  communities?. . . People  began  to  be  frustrated  about 
what  the  state's  vision  of  local  control  and  community 
planning  was.  Are  there  limits  to  what  you  can  do?  Are 
there  expectations  from  the  state  regarding  goals  and 
objectives?  I think  there  are  very  definite  expectations  they 
want  to  see  met,  but  they've  not  communicated  that  to  us." 

At  the  intrateam  level,  effective  communication  was  also 
identified  as  a predictor  of  group  cohesion.  Participants 
revealed: 

"I  think  that  there  are  a couple  of  primary  players. . . I think 
that  those  people  need  to  let  go  a little  bit  and  communicate 
better  with  our  team  in  terms  of  [the  information]  that  the 
state  gives.  A lot  of  times  when  we  go  to  the  meetings  there 
are  two  or  three  people  who  really  Imow  what's  been  sent 
and  really  have  a handle  on  what's  going  on.  They're  not 
really  good  at  getting  that  information  disseminated  to  the 
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really  good  at  getting  that  information  disseminated  to  the 
rest  of  the  group. . . I don't  know  whether  it's  because  they 
need  to  control  the  information  or  they  just  need  to  make 
things  happen.  Sometimes  it's  easier  to  say  'Oh  I'll  do  it 
myself  because  that's  a quicker  way  to  m^e  things  happen.' 
But,  it  doesn't  do  the  team  building  piece." 

This  observation  was  confirmed  by  other  respondents: 

"It  seems  that  by  default. . .that  [one  person]  has  access  to  a 
lot  of  information.  It  is  that  [person's]  interpretation  of 
information  that  we  get  and  I do  feel  that  that  is  not 
appropriate.  I believe  that  as  team  members  we  should  all 
have  access  to  the  same  information." 

"I  think  dissemination  has  been  a problem.  Since  there  isn't 
a coordinator  for  the  team,  a member  of  the  team  who 
already  has  another  job  is  getting  the  information  and  then 
has  to  fit  in  time  to  mail  out  sometimes  rather  large  volumes 
of  paper.  So,  it  doesn't  always  get  out  in  as  timely  a manner 
as  would  be  helpful." 

5.  Inclusive  Community  Representation 

Part  of  the  Access  Vermont  Initiative  calls  for 
representation  reflective  of  the  local  catchment  area 
population.  Mattessich  and  Monsey  (1993)  recommend  that 
"the  collaborative  group  includes  representatives  from  each 
segment  of  the  community  who  will  be  affected  by  its 
activities".  This  theme  was  prominent,  poignant,  and 
recurrent. 

"I  think  that  teams  have  to  like  to  work  with  parents.  . . 
Specialists  seem  to  think  that  they  give  the  service  so  that 
they  know  more  about  things.  Most  parents  feel  inferior.  . . 
some  committees  only  want  parents'  signatures  to  show  that 
they're  trying  to  work  with  parents,  but  they  don't  really  care 
what  parents  think.  Professionals  [need  to]  become  human. . 
. to  put  themselves  in  the  parents  shoes  and  not  just  look  at 
their  books.  Parents'  voices  are  valuableE  they  have  to  listen 
to  parents  and  make  them  feel  a part  of  the  team  not  make 
them  feel  they  have  to  be  there  because  that's  how  they  get 
grants.  So,  they  don't  really  want  the  parents'  participation, 
but  the  state  says,  'You  really  should  have  parents;  you 
should  listen  to  what  parents  want  and  stuff  How's  that  for 
honesty?" 

For  their  part,  professionals  are  beginning  to  recognize  that 
team  environments  can  be  uninviting  and  threatening  to 
non-professionals.  One  agency  member  reflects: 

"It's  been  a real  difficult  test  [inclusion  of  parents]  and  we 
still  have  a lot  more  to  do  to  make  this  process  more 
attractive  to  families.  We  have  to  create  an  environment 
where  youth  and  family  people  feel  safe  coming  to  a 
meeting.  I feel  uncomfortable  in  a room  full  of  bureaucrats, 
and  I think  we  have  to  be  sensitive  to  what  that  feels  like  to 
families." 
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Although  there  appeared  to  be  a general  consensus 
regarding  the  need  and  desire  for  increased  representation,  it 
also  appeared  that  the  achievement  of  this  goal  will  require 
significant  and  sensitive  commitment  from  involved 
professionals.  Recognition  of  the  complexities  associated 
with  more  inclusive  representation  have  perhaps  worked  to 
delay  realization  of  this  goal.  Team  dynamics  might  be 
predicted  to  change  as  new  members  come  on  board,  and  it 
would  be  necessary  for  the  language  of  the  team  to  be 
accessible  to  all.  One  professional  contended: 

"There  are  not  enough  parents  in  the  group.  I think  everyone 
agrees  that  there  needs  to  be  more  people,  but  it  will  be 
interesting  to  see  how  the  dynamic  changes.  In  the  long  run, 
I think  we  can  function  without  more  community 
representation,  but  certainly  we'd  get  a broader-based  view 
when  we  add  [people].  You  know,  you  bring  in  different 
people  from  different  areas  and  they  have  different 
perspectives." 

A parent  reflected; 

"It  was  like  I was  taking  up  too  much  of  their  time.  Before 
you  can  make  a comment  about  something,  you  have  to 
know  what  they're  talking  about.  They  did  not  want  the 
bother  of  explaining  to  a parent  that  had  been  through  just 
part  of  the  system. . . I think  the  state  level  is  very  high  on 
encouraging  parents  to  participate. . . Instead  of  giving 
trainings  to  the  parent  all  the  time,  give  some  to  the 
professionals.  ^Tiat  does  it  mean  to  have  a parent  in  the 
group?  What  do  you  expect  from  a parent?" 

We  summarized  the  essence  of  these  comments  with  the 
recognition  that  these  teams  are  in  the  early  stages  of 
development  and  will  inevitably  evolve  over  time.  In  the 
words  of  one  member: 

"Two  years  from  now  it  may  not  look  like  what  it  does  now. 
That's  kind  of  the  exciting  part  of  the  whole  thing.  So,  we 
just  keep  moving." 

Conclusion 

The  purpose  of  this  project  was  to  explore  the  development 
of  two  Vermont  Stakeholders'  Teams  formed  to  respond  to 
the  Access  Vermont  Initiative.  The  researchers  had  hoped  to 
draw  from  this  developmental  account  an  understanding  of 
the  dynamics  associated  with  collaborative  efforts.  It  was 
proposed  that  identification  of  variables  influencing  the 
team  process  might  serve  as  a tool  for  the  development  of  a 
learning  model  or  source  of  reference  for  future  teams. 
Awareness  of  the  multiple  variables  influencing  a 
collaborative  team  process  and  dynamic  is  critical  to 
opening  the  process  to  potential  mediation  and  consequent 
modification.  This  enhanced  awareness  may  also  allow  a 
team  to  move  beyond  temporary  obstacles  inhibiting 
effective  collaboration  to  a realization  of  its  full  and  most 
productive  potential. 


productive  potential. 


Although  five  predominant  themes  were  identified  from  the 
data,  the  individual  themes  themselves  appear  to  call  for 
continued  investigation.  Future  research  questions  might 
include  &endash;  "What  are  the  precursors  or  antecedents 
leading  to  the  establishment  of  trust  in  intrateam 
configurations?"  and  "How  are  intrateam  trusting 
relationships  maintained?"  Future  studies  might  also 
investigate  the  effect  of  crises  such  as  economic  change  on 
team  dynamic.  The  movement  toward  integration  and 
inclusion  of  non-professional  participants  also  calls  for 
exploration.  Both  professional  and  parent  representatives 
identified  parent  membership  in  these  two  teams  as 
problematic,  and  future  studies  should  address  clear 
definitions  of  the  specific  barriers  to  parental  involvement. 
Researchers  speculate  the  team  members'  difficulty  in 
articulating  these  barriers  may  reflect  the  teams'  early  stage 
of  development.  Many  participants,  both  professional  and 
non-professional,  identified  increased  parent  participation 
and  non-professional  involvement  on  teams  as  critical  to 
future  team  success.  Members  offered  thoughts  regarding 
ways  to  increase  inclusion  and  resulting  cohesion;  these 
included  provision  of  opportunities  for  parent  networking 
and  professional  training  highlighting  the  value  of 
non-professional  resources.  A clear  identification  of 
obstacles  to  inclusive  representation  might  aid  the  design  of 
such  training  structures. 

This  study  is  limited  in  that  only  two  Vermont  teams  were 
included  in  the  analysis,  and  in  that  these  teams  are  truly  in 
the  early  stages  of  development;  their  current  status  may  not 
reflect  foture  developmental  patterns.  However,  research 
designed  to  follow  these  and  other  teams  longitudinally  may 
prove  revealing. 

The  value  of  this  study  lies  with  its  contribution  to  the 
literature  on  team  process.  Future  directions  might  lead  to 
the  development  of  a quantitative  instrument  designed  to 
measure  process  and  progress  in  interdisciplinary  team 
evolution.  Ongoing  qualitative  work  will  serve  to  inform 
future  quantitative  research  with  regard  to  issues  relevant  to 
productive  team  functioning. 
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Appendix  1 

Stakeholders'  Team  Interview  Questions 

1 . What  do  you  see  as  your  role  as  a working  team  member? 

2.  What  are  your  personal  reasons  for  involvement  with  this 
team? 

3.  What  do  you  see  as  the  purpose  of  this  team?  How  is  (or 
is  not)  the  team  fulfilling  this  purpose  or  goal? 

4.  What  are  the  personal  rewards  or  benefits  of  affiliation 
with  the  team?  How  does  your  target  population  (children 
with  serious  emotional  disabilities  and  their  families) 
benefit  from  the  presence  of  this  team  in  your  community? 

5.  From  your  perspective,  does  your  team  provide  for  a fair 
distribution  of  responsibilities,  roles,  resources,  and  credit 
for  accomplishments?  Is  its  structure  hierarchical  or 
egalitarian?  Are  there  defined  leaders  on  the  team  or  is 
leadership  shared? 

6.  Do  you  feel  that  your  team  was  given  guidance  or 
direction  related  to  the  formation  of  a working, 
collaborative  team?  If  not,  do  you  believe  guidance  would 
enhance  team  function?  If  so,  what  was  the  nature  of  the 
guidance  or  intervention?  Did  this  guidance  or  intervention 
affect  your  team's  collaborative  effort? 

7.  Have  there  been  obstacles  or  barriers  realized  in  the 
process  of  building  the  team?  From  your  perspective,  what 
has  been  the  most  significant  obstacle  to  team  development? 
How  have  team  members  worked  to  address  and  surmoimt 
these  obstacles? 


8.  Can  you  identify  problems  or  conflicts  that  your  team  has 
encountered?  When  problems  or  conflicts  have  arisen,  how 
has  your  team  worked  to  resolve  them?  Does  the  team 
address  the  problem  directly  and  openly  or  does  it  appear 
that  it  is  the  nature  of  the  team  to  leave  the  problem  or 
conflict  unaddressed? 

9.  Have  you  experienced  or  recognized  competitive  or 
territorial  attitudes  among  or  between  members  of  your 
team?  If  so,  in  your  view,  how  have  these  attitudes  affected 
team  function?  How  do  you  personally  deal  with 
territoriality? 

10.  Do  you  feel  that  each  team  member  is  respected  or 
valued  and  her/his  voice  heard  at  meetings? 

1 1 . Can  you  identify  any  issues,  events,  or  situations  that 
have  contributed  to  team  cohesion  and  cooperation  among 
members? 

12.  In  your  view,  how  can  collaboration  be  improved?  What 
actions  might  be  taken  to  enhance  team  unity  and 
cooperative  spirit  among  and  between  team  members? 

1 3 . How  can  individuals  or  collaboratives  at  the  state  level 
facilitate  interagency  collaboration  at  the  local  level? 

14.  Do  team  members  appear  to  have  a common  vision  for 
improved  services  to  severely  emotionally  disturbed 
children  and  adolescents  and  their  families? 

15.  From  your  perspective,  do  you  feel  the  team  works 
effectively?  Do  you  see  yourself  as  part  of  a team? 
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Introduction 
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Children  in  the  education  system  with  complex  needs,  at 
risk  of  costly  institutionalization,  present  natural 
opportunities  for  cost-shared,  collaborative  planning,  and 
service  coordination.  Many  of  these  students  may  also  be 
children  served  by  the  child  welfare  system,  office  of  mental 
health  services,  and/or  the  juvenile  justice  system. 


During  the  1979-80  school  year,  Florida's  public  schools 
contracted  for  72  residential  placements  of  students  with 
severe  emotional  disturbances;  during  the  1980-81  school 
year,  83  such  placements  were  made  prior  to  February  of 
1981.  Total  contract  costs  for  these  placements  ranged  from 
about  $1 ,000  to  over  $248,000  per  student.  As  a result  of  the 
increase  in  contracted  services  for  residential  placements  by 
public  school  systems  and  the  identified  need  for 
community  based  mental  health  services,  the  Bureau  of 
Education  for  Exceptional  Students,  Division  of  Public 
Schools,  Department  of  Education  (DOE)  was  given  the 
responsibility  for  conducting  a study  to  determine  the  need 
for  services  for  students  with  severe  emotional  disturbance 
(DOE,  1981).  In  February,  1981,  a Report  of  the  Study  to 
Determine  the  Need  for  Educational  Centers  for 
Emotionally  Disturbed  Students  was  prepared  for  the 
Commissioner  of  Education.  This  study  provided  the 
information  essential  to  the  Florida  Legislature  to  develop 
the  Multiagency  Network  authorizing  legislation. 


The  study  found  that  the  majority  of  children  who  were  in 
state-funded  residential  treatment  were  being  served  in 
programs  outside  the  State  of  Florida,  and  that  75%  of 
Florida's  children's  mental  health  funds  were  being  spent  on 
residential  treatment.  Very  few  exceptional  student 
education  programs  existed  for  students  with  severe 
emotional  disturbance.  There  was  only  minimal 
community-based  multiagency  service  planning, 
coordination,  or  delivery.  Consequently,  children's  mental 
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health  services  were  inadequate  with  virtually  nothing  to  fill 
the  void  between  outpatient  care  and  hospitalization.  It  was 
also  determined  that  the  family  involvement  in  children's 
mental  health  planning  and  programs  was  minimal  or 
non-existent,  and  communication  was  limited  across 
programs  or  agencies.  There  was  limited  cross  program 
training,  and  any  efforts  in  assessment  were  redundant  and 
not  focused.  Finally,  there  was  an  absence  of  information 
and  referral  services,  crisis  intervention  services, 
school-based  mental  health  services,  and  multiagency  case 
management  for  families. 

Method 

To  address  this  multi-agency  service  issue,  appointments  by 
the  Secretary  of  Health  and  Rehabilitative  Services  and  the 
Commissioner  of  Education  created  a State  Advisory  Board 
composed  of  leaders  in  education,  children's  mental  health, 
and  children's  advocacy.  Working  in  an  advisory  capacity  to 
the  Bureau  of  Student  Services  and  Exceptional  Education, 
a plan  to  offer  fiscal  incentive  for  collaboration  was 
developed  and  made  available  to  select  school  districts 
within  each  region  of  the  Department  of  Health  and 
Rehabilitative  Services  (DHRS). 

SEDNET  projects  were  awarded  to  one  school  district 
within  each  DHRS  region,  and  required  signed  cooperative 
agreements  between  DHRS  and  each  school  district.  In 
many  regions,  projects  were  funded  through  the  Individuals 
with  Disabilities  Education  Act  (IDEA)  Part  B,  General 
Revenue,  and  local  contributions.  Each  project  was  staffed 
by  one  full  time  project  manager  and  various  support  staff. 
To  date,  over  five  hundred  leaders  in  education,  health  and 
human  service  systems,  mental  health  centers,  parent 
coalitions,  and  juvenile  justice  entities  voluntarily  serve  on 
regional  SEDNET  Advisory  Boards. 

Florida  initially  developed  an  urban  and  rural  model  in 
1982-83.  Pilot  sites  demonstrated  the  need  for  local 
flexibility  and  control  for  maximum  success.  Therefore,  a 
statewide  framework  was  developed  requiring 
locally-determined,  needs-based  priorities  consistent  with 
guiding  principles.  Statewide  network  goals  included  the 
following:  (a)  multiagency  planning  for  a complete  array  of 
services,  (b)  continuous  improvement  of  service  in  the 
system  of  care,  (c)  evaluating  the  effectiveness  of  the 
system,  and  (d)  disseminating  information  regarding  the 
system  of  care. 

In  1985,  DOE  contracted  with  the  Department  of  Child  and 
Family  Studies,  Florida  Mental  Health  Institute,  University 
of  South  Florida,  to  administer  the  Quality  Development 
Teams  (QDT),  in  order  to  identify  characteristics  of 
successful  projects  and  evaluate  SED  Network  Project 
impact  on  the  delivery  and  effectiveness  or  services  to 
children  with  SED.  Statewide  project  reviews  were 
conducted  and  features  of  quality  projects  and  effective 
networks  with  successful  outcomes  were  examined  at  the 
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networks  with  successful  outcomes  were  examined  at  the 
planning,  implementation,  and  evaluation  levels. 
Multidisciplinary  teams  looked  at  what  enabled  networks  to 
bring  their  children  back  to  their  communities  from  out  of 
state  and  county  placements,  create  new  special  education 
programs  and  day  treatment  programs,  intervene  in  times  of 
crises,  provide  multi-agency  case  planning  and  case 
management,  and  have  meaningful  parent  involvement. 

Standards  and  self  assessment  tools  were  developed  for  both 
the  projects  and  networks,  in  addition  to  identification  of 
competencies  of  effective  project  managers,  board 
members,  and  other  network  participants.  Five  volumes  of 
best  practices  were  published  as  additional  tools  to  assist 
communities  in  creative  problem-solving,  and  program  and 
professional  development. 

Florida's  legislation  authorizing  the  SED  Network  was 
amended  in  1 990  to  include  a requirement  for  annual 
reporting  to  the  Legislature.  Four  annual  reports  have  been 
produced  to  date,  summarizing  regional  and  statewide 
accomplishments  and  providing  recommendations  for 
further  progress. 

Results 

Although  enrollment  in  schools  has  grown  by  32  percent 
over  the  past  nine  years,  according  to  the  March  2,  1 995 
memorandum  to  the  Senate  and  House  Appropriations 
Committee  1,  only  28  students  with  severe  emotional 
disturbance  were  in  private  residential  programs  under 
school  district  contracts  during  fiscal  year  1994-95.  This  is  a 
reduction  of  61%.  Similar  progress  is  evident  in  DHRS 
placement  data,  with  a significant  increase  in  Florida's 
ability  to  serve  children  in  therapeutic  foster  homes  versus 
more  costly  residential  treatment. 

Florida's  progress  over  the  last  ten  years  is  also  evidenced  in 
outcomes  of  students  with  severe  emotional  disturbance. 
According  to  enrollment,  placement,  and  exit  data,  there  has 
been  a 60%  increase  in  the  numbers  of  students  with  severe 
emotional  disturbance  graduating  from  high  school.  Also, 
there  has  been  a 25%  increase  of  students  identified  as 
eligible  for  educational  programs  for  youth  with  emotional 
disabilities,  albeit  consistent  with  the  increase  in  Florida's 
student  population.  There  are  1 6%  fewer  students  with  SED 
being  placed  outside  the  public  school  system,  and  a total  of 
15%  more  students  with  emotional  disabilities  are  being 
served  in  regular  education  classrooms.  Finally,  there  has 
been  a 100%  decrease  in  the  number  of  students  who  are 
placed  outside  the  state  for  residential  treatment  with 
children's  mental  health  funds  and  an  estimated  3,000 
students  were  diverted  from  residential  treatment  during  the 
past  fiscal  year  through  DHRS  Family  Service  Planning 
Teams. 

In  1995,  families  were  more  involved  and  active  with 
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multiagency  planning  at  every  level  than  ever  before.  Due 
mainly  to  strong,  cohesive  groups  and  organizations,  parents 
had  opportunities  to  be  active  partners  in  planning  and 
service  delivery.  This  has  resulted  in  policy  enhancements 
and  changes  that  impact  favorably  on  Florida's  students  with 
severe  emotional  disturbance.  The  SED  Network  has 
assisted  in  the  growth  and  development  of  organizations, 
such  as  The  Florida  Alliance  for  the  Mentally  111  - Child  and 
Adolescent  Network  (FAMI-CAN);  The  Florida  Federation 
of  Families  for  Children's  Mental  Health  (FFFCMH);  and 
The  United  Families  for  Children's  Mental  Health 
(UFCMH). 

Network  projects  now  boast  100%  family  attendance  and 
participation  in  Family  Service  Planning  Team  operations, 
where  multidisciplinary  case  plans  are  developed,  case 
managers  are  designated,  and  integrated  service  plans  are 
agreed  upon  by  the  significant  participants  in  each  child's 
life.  In  addition,  family  members  serve  on  the  State 
Advisory  Board  and  on  17  of  the  18  regional,  multiagency 
advisory  boards. 

Discussion 

By  providing  staff  support,  the  DOE  creates  opportunities 
for  school  districts  to  unite  in  a regional  structure  consistent 
with  the  DHRS,  fiscal  agents  for  children's  mental  health 
funds.  This  approach  has  resulted  in  a rich  diversity  of 
decentralized  structures  and  approaches,  reflecting  local 
decision-making  that  builds  on  community  leadership  and 
resources.  This  structure  also  allows  for  the  maximization  of 
scarce  resources,  supplementing  the  capability  of  a single 
county  school  district. 

According  to  the  Florida  Children's  Mental  Health  Plan  ( 
annually  required  for  federal  block  grant  funds),  $228 
million  was  spent  on  children's  residential  treatment 
services  in  1995-96.  The  benefits  of  these  funds  is 
maximized  with  state,  regional  and  locally  integrated 
planning,  implementation  and  evaluation.  The  total  cost  of 
the  Statewide  SED  Network  activities  which  assist  with  this 
function,  as  well  as  addressing  the  goal  of  an  integrated 
system  of  care,  is  2.2  million  annually.  During  the  1995-96 
legislature  session,  a 190%  increase  in  General  Revenue 
was  allocated  for  SEDNET.  As  a result  of  an  increased 
awareness  for  the  needs  of  youth  with  emotional  disabilities, 
advocacy,  and  research  demonstrating  the  effectiveness  of 
early  intervention,  Florida  has  (a)  increased  its  budget  for 
children's  mental  health  by  one  million  dollars  in  the  last 
decade,  (b)  increased  the  number  of  children  served  by  78%, 
(c)  reduced  the  expenditures  per  child  by  73%,  and  (d) 
decreased  the  number  of  residential  treatment  from  29.3 
days  per  recipient,  in  1992,  to  14  days,  in  1994  or  52%. 

The  role  of  the  state  is  to  provide  statewide  coordination, 
identify  and  replicate  effective  practices,  and  provide 
technical  assistance  and  eval-uation.  Multiagency 
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networking  contributes  to  student  graduation,  increased 
recognition  of  program  and  service  eligibility,  greater 
inclusion,  improved  access  and  effectiveness  of  the  system 
of  care,  family  involvement,  and  a broader  array  of 
community-based  education  and  treatment  alternatives. 
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Introduction 


In  Florida,  Case  Review  Committees  (CRCs)  were 
developed  in  the  early  1980's  to  be  the  gatekeepers  of 
residential  placement  for  children  with  emotional 
disabilities.  There  were,  however,  over  400  children 
statewide  on  CRC  waiting  lists,  and  no  identified  planning 
mechanism  to  maintain  those  children  in  the  community 
awaiting  placement  or  to  divert  children  from  CRCs.  The 
Family  Service  Planning  Teams  (FSPT)  were  initiated  in 
1987  by  the  Office  of  Alcohol,  Drug  Abuse  and  Mental 
Health,  Department  of  Health  and  Rehabilitative  Services 
(DHRS)  to  accomplish  this  purpose. 

FSPTs  were  implemented  not  as  a result  of  new  funding,  but 
from  a commitment  of  DHRS  and  the  Department  of 
Education  (DOE)  to  create  interagency  planning  forums  to 
develop  holistic  service  plans  to  enable  children  to  live  in 
the  community  and  be  successful  in  school.  The 
Multiagency  Network  for  Students  with  Severe  Emotional 
Disturbance  (SEDNET)  was  established  by  the  Florida 
Legislature  in  1981  in  recognition  of  the  significant  cost  and 
lack  of  progress  of  students  served  outside  the  state  in 
psychiatric  residential  treatment.  SEDNET  project 
managers,  already  functioning  as  regional  network 
coordinators  throughout  Florida  and  funded  by  DOE, 
became  the  facilitators  of  this  new  planning  effort.  FSPTs 
have  become  the  focus  of  service  planning  for  children  at 
risk  of  out-of-home  or  school  placement  in  need  of 
multiagency  involvement,  bringing  together  parents  and 
social  service  and  education  professionals. 

Since  implementation  of  FSPTs,  many  of  the  children 
previously  in  out-of-state  residential  treatment  have  been 
able  to  return  to  Florida.  Referrals  to  CRCs  have  decreased. 
Concurrent  educational  program  development  and 
multiagency  cooperative  systems  of  support  have  promoted 
educational  reviews  of  clients  of  DHRS  resulting  in  a 260% 
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increase  in  student  eligibility  and  in  services  provided 
through  educational  programs  for  students  with  severe 
emotional  disturbance.  Multiagency  case  management 
services  have  simultaneously  increased,  as  have 
parent/professional  partnerships,  access  to  Medicaid,  and 
private  contributions  to  local  systems  of  care. 

In  response  to  continuing  needs  for  improving 
implementation,  identifying  training  needs,  and  supplying 
additional  data  on  this  pre-residential  intervention,  a 
multidisciplinary  plaiming  team  review  was  conducted. 

Method 

The  multidisciplinary  planning  team  review  was  conducted 
in  three  parts.  Part  One  focused  on  the  utilization  of  services 
by  the  FSPTs.  SEDNET  Project  Managers  were  asked, 
"What  services  purchased  by  the  FSPT(s)  make  the  most 
difference  in  improving  the  lives  of  the  children  and 
families  served?" 

Part  Two  provided  a status  report  of  the  children  whose 
collaborative  treatment  plans  were  described  in  one  of  four 
Best  Practices  in  Creative  Service  Delivery  manuals 
(1989-1992).  A total  of  forty-seven  cases  were  selected. 
Project  Managers  were  asked,  "What  was  the  major 
contributing  factor  of  successful  service  plaiming?"  and 
"What  was  the  most  valuable  contribution  of  the  system  to 
this  child?" 

Part  Three  focused  on  a single  county's  FSPT  efforts  to 
assess  the  overall  satisfaction  of  the  consumers  they  served. 
This  FSPT  also  conducted  a needs  survey  to  identify  those 
services  most  frequently  requested  by  parents/guardians. 
Twenty  interviews  with  parents/guardians  were  conducted. 

Results 

Part  One 

Responses  were  received  from  14  of  18  regional  SEDNET 
projects,  representing  50  of  Florida's  67  counties  (75%). 
Ranked  in  the  order  of  making  the  most  difference, 
purchased  services  included: 

1 . Recreational/leisure  activities  aimed  at  improving 
self-esteem  & social  skill  development. 

2.  In-home  services/in-home  parent  education  and 
counseling/therapy. 

3.  Respite  care 

4.  Behavior  Specialist/Certified  Behavior  Analyst  in  school 
and  home 

5.  Mentor/Therapeutic  friends 
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5.  Mentor/Therapeutic  friends 

6.  Case  Management 

Part  Two 

Looking  at  Family  Service  Plan  development  process  and 
outcome,  SEDNET  was  able  to  provide  the  child's  status 
and  residential  placement  history  for  sixteen  of  the 
forty-seven  children.  (Children  in  the  Best  Practices  manual 
were  identified  by  numbers  or  initials  only.  Identifications 
relied  on  project  manager's  or  case  manager's  memory  of  the 
child  from  brief,  anecdotal  case  presentations.)  Of  these  16, 
10  of  the  youth  staffed  at  FSPT  (63%)  did  not  go  on  to  the 
CRC,  although  three  were  placed  in  residential  facilities. 
Four  youths  did  go  on  to  CRC;  three  were  placed  in 
residential  facilities,  and  one  remained  at  home  with  support 
services.  Two  of  the  sixteen  youths  reviewed  had  already 
been  discharged  from  residential  placement  and  went  to 
CRC  for  review  only. 

In  response  to  the  question,  "What  was  the  major 
contributing  factor  of  successful  service  planning?," 
findings  of  the  reviews  suggest  that  when  the  youth  did  not 
go  on  to  CRC,  there  was  a consistent  individual  involved  in 
planning  and  implementation  of  service  plans.  Involvement 
was  primarily  with  a family  member,  usually  the  mother. 
Other  individuals  included  biological  father,  case  manager, 
and  teacher  in  an  SED  classroom.  Consistent  family 
involvement  was  also  the  common  factor  with  the  one  child 
who  went  to  CRC  but  was  not  subsequently  placed 
out-of-home. 

With  the  three  youths  who  were  placed  in  residential 
treatment  through  the  CRC,  family  involvement  was 
described  as  less  consistent:  (a)  one  child  had  family 
involvement  in  the  early  years  but  less  involvement  after 
placement;  (b)  one  child  with  the  mother  involved,  but  her 
own  addictions  problematic  to  consistent  treatment;  and  (c) 
the  third  child  with  parents  who  were  divorced  but  involved 
and  concerned.  In  the  two  cases  where  the  youths  were 
reviewed  at  the  CRC,  following  discharge  from  residential 
placement,  one  family  was  involved  in  team  planning  from 
the  beginning  with  regular  communication  with  the  case 
manager,  while  the  other  family  became  more  involved  and 
participatory  as  they  began  to  see  improvement.  The  three 
youths  who  did  not  go  to  CRC  but  were  placed  in  residential 
treatment  were  youths  in  foster  care  with  no  direct  family 
involvement. 

When  asked  "What  was  the  most  valuable  contribution  of 
the  system  to  this  child?,"  the  three  most  frequent  responses 
were:  (a)  looking  at  the  family  as  a unit,  (b)  keeping  the 
child  in  the  home  with  in-home  services,  and  (c) 
multiagency  planning  meetings. 

Part  Three 

The  FSPT  in  Hernando  County  endeavored  to  assess  the 
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The  FSPT  in  Hernando  County  endeavored  to  assess  the 
overall  satisfaction  of  the  consumers  whom  they  served. 
These  consumers  were  determined  by  the  committee  to  be 
both  the  children  of  concern  and  their  parents/guardians. 
However,  only  the  parents/guardians  were  surveyed. 

The  FSPT  Satisfaction  Survey  consisted  of  29  statements 
based  on  a 5 point  Likert-type  scale  to  determine  their 
satisfaction  and  attitudes  toward  the  FSPT  process  and 
actual  services  provided. 

• 74%  felt  included  in  the  planning  of  services  at  the 
FSPT,  wanted  to  come  to  the  meeting,  and  be  part  of 
the  process. 

• 58-73%  felt  they  were  treated  with  respect  and  given 
the  opportunity  to  speak. 

• 64-85%  felt  that  their  beliefs,  values,  and  points  of 
view  were  used  to  plan  services. 

• 74-95%  felt  that  FSPT  provided  good  services  for 
children/families. 

• 74-95%  felt  that  the  goals  of  the  Family  Service  Plan 
were  explained  adequately. 

• 36%  felt  blamed  for  their  child's  problems. 

• 43%  strongly  agreed,  and  26%  somewhat  agreed  that 
their  children  could  be  helped  in  the  home. 

• 63%  did  not  feel  intimidated  by  the'process. 

• 69%  felt  that  the  FSPT  committee  used  the  family's 
day-to-day  life  to  plan  for  services. 

Recommendations  based  on  survey  outcomes  include;  (a) 
hiring  a Parent  Liaison  to  help  parents  feel  less  intimidated 
with  the  FSPT  process  and  be  more  of  an  integral  member 
of  the  case  plan;  (b)  developing  strategies  to  overcome  the 
transportation  barrier  and  provide  more  in-home  services 
whenever  possible;  (c)  developing  surveys  for  the  children 
of  concern  so  that  their  satisfaction  can  also  be  measured; 
and  (d)  including  the  children  of  concern  in  the  FSPT 
process  whenever  possible. 

The  needs  survey  to  determine  what  services  were  most 
frequently  requested  by  parents/guardians  consisted  of 
twelve  individual  categories  of  services  and  was  measured 
on  a 0 - 2 scale. 

• 70%  felt  the  following  services  were  very  needed: 
Case  management,  recreational  activities,  one-to-one 
adult/child  relationship  (e.g..  Big  Brother/Big  Sister). 

• 50-65%  felt  the  following  services  were  very  needed 
(ranked  in  order  of  importance):  (1)  Financial 
assistance;  (2)  counseling,  family,  individual  and 
group;  (3)  social  skills  training;  (4)  evaluations;  (5) 
vocational  training/job  coach;  and  (6)  home-based  ' 
services. 

• 60%  felt  out-of-the-community  residential  services 
were  not  needed. 

• 45%  felt  parent  education  was  very  needed. 

• 45%  felt  parent  education  was  somewhat  needed. 
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Recommendations  based  on  the  results  of  the  Needs  Survey 
included:  (a)  developing  a therapeutic  mentorship  program; 
(b)  provision  of  financial  support  for  a child  psychiatrist  so 
that  medication  management,  therapy,  case  management, 
and  non-clinical  services  can  be  obtained  for  Medicaid 
clients;  (c)  continued  parent  education  in  the  community; 

(d)  increased  respite  in  the  community  for  special  needs  and 
at-risk  children;  and  (e)  continued  funding  for  specialized 
training/personnel  for  summer  recreation  programs. 

Discussion 

Multiagency  case  planning  has  allowed  concerned 
individuals  to  request  that  significant  partners  in  a child's 
life  collaborate  in  the  development  and  implementation  of 
the  best  possible  plan  utilizing  existing  community 
resources,  or  with  limited  availability,  new  resources.  The 
family-centered,  customized  approach  to  service,  resource 
development  and  coordination  on  behalf  of  children  who 
need  services  from  multiple  systems  has  been  key  to  the 
success  of  regionally-initiated  FSPTs.  Multiagency  case 
planning  has  also  provided  continuous  needs  assessment 
data  to  program  plarmers  for  improving  the  local  system  of 
care.  With  resources  at  a premium,  identifying  and 
providing  the  services  which  make  the  most  difference  in 
efforts  to  assist  youth  with  special  needs  is  more  important 
now  than  ever  before. 

The  key  services  identified  in  this  study  reflect  the  concerns 
of  this  special  population.  For  example,  many  students  lack 
initiative,  self-discipline,  and  continuous 
guidance^upervision,  resulting  in  the  need  for  external 
support  systems  to  boost  self-esteem,  as  well  as  to  increase 
their  ability  to  seek  out  constructive  activities  for  personal 
growth.  Similarly,  the  ability  to  provide  essential  respite 
care  at  the  time  of  need  has  long  been  reported  as  a rnajor 
contributor  to  parental  ability  to  maintain  children  with 
intense  needs  within  their  homes.  Traditional  outpatient 
services  often  neglect  the  ability  and  potential  of  the 
caretaker  in  shaping  behaviors  and  providing  extended 
support.  Counseling  services  made  available  to  children  and 
parents  in  their  home  setting  may  allow  a more  complete 
view  of  family  and  environmental  dynamics,  and  contribute 
to  more  accessible  and  effective  interventions.  Additionally, 
the  study  shows  that  parents,  teachers,  and  caretakers  of 
children  often  lack  the  benefit  of  professionally  trained 
experts  in  systemic  approaches  to  behavior  management. 
The  services  of  specialists  in  behavior  management, 
however,  not  only  provide  direct  assistance  with  student  and 
child-behaviors,  but  provide  a source  of  instruction  and 
technical  assistance  to  teachers/caretakers.  Also,  the  value 
of  a significant  other  has  been  consistently  documented  in 
mental  health  and  delinquency-related  research;  the  use  of 
mentors  or  therapeutic  friends,  in  volunteer,  paid,  or 
Medicaid  ftmded  non-clinical  in  home  service  utilization  is 
perceived  as  having  made  a difference  in  positive  outcomes. 
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Finally,  designating  a service  coordinator  as  needed  is  a key 
to  success  in  assisting  and  monitoring  the  implementation  of 
many  multiagency  plans. 

New  funds  earmarked  for  student  success  and  support  are 
available  through  delinquency  prevention  opportunities, 
family  preservation,  family  support,  and  other  initiatives 
with  similar  missions.  Managers  of  these  funds  and 
prospective  providers  are  encouraged  to  maintain  and/or 
develop  responsive  programs,  new  and  creative  services, 
and  maximize  opportunities  for  flexibility  necessary  for 
truly  individualized  services. 
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Introduction 


The  Prime  Time  Project  is  a comprehensive  intervention 
model  for  youths  who  are  at  the  extreme  end  of  the  juvenile 
justice  continuum  due  to  their  history  of  repeat  offending, 
relatively  severe  offenses,  a high  degree  of  isolation  or 
abandonment  from  their  natural  community,  and  the 
co-occurrence  of  mental  health  disorder.  Prime  Time  was 
initiated  in  the  fall  of  1995  in  King  County, 

Washington.  Faced  with  rising  juvenile  crime  and  violence, 
increasing  public  concern,  and  a corresponding  rise  in 
demands  placed  on  the  county  juvenile  justice  system, 
county  officials  have  voiced  a high  level  of  interest  in 
alternative,  or  enhanced,  services  to  address  juvenile  crime. 
The  county  council  awarded  a two-year  grant,  recognizing 
that  effective  interventions  exist  and  that  linkages  and 
collaboration  between  agencies  and  service  providers  play  a 
role  in  successful  intervention.  The  goals  negotiated  with 
the  county  were  simple:  (a)  reduce  recidivism,  (b)  reduce 
severity  of  violent  and  other  offending  behavior,  (c)  increase 
school  attendance  and  performance,  and  (d)  increase 
residential  stability.  The  county  council  also  mandated  that 
we  target  those  youth  in  the  "deep  end"  of  the  juvenile 
offender  population.  This  translated  to  the  following 
entrance  criteria  for  youth  in  the  program:  (a)  currently  in 
detention;  (b)  age  12  to  17;  (c)  at  least  two  admissions  to 
detention;  (d)  adjudicated  for  a relatively  serious  offense 
(e.g.,  assault,  burglary,  vehicular  homicide,  etc.);  and  (e) 
presence  of  a diagnosable  mental  health  disorder. 

Theoretical  Background 

Social-ecological  models  explain  delinquent  behavior  as 
multiply  determined  by  interactions  between  genetic. 
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psychiatric  or  psychological  variables,  and  key  elements  of 
the  environment  such  as  family,  school,  peers,  and  other 
systems.  Accepting  this  approach,  we  found  more  specific 
guidance  in  Patterson's  (1982)  explication  of  a "coercive 
cycle"  in  aggressive  children's  antisocial  interactions. 
Patterson  described  an  interaction  pattern  in  which  a youth's 
antisocial  behavior  is  followed  by  the  parent's  negative 
reactions.  This,  in  turn,  escalates  the  youth's  antisocial, 
aggressive  behavior,  triggering  a cycle  that  is  both  cause 
and  effect. 

With  Patterson's  model  in  mind,  we  noticed  a striking 
pattern  of  coercive  exchanges  across  a wide  range  of 
systems  in  which  juvenile  offenders  interact.  For  many  of 
these  youths,  reciprocal  coercive  exchanges  characterize 
interactions  with  family  members,  teachers  and  other  school 
officials,  and  members  of  the  community.  Once  referred  to 
the  juvenile  justice  system,  the  youth's  interactions  with 
police,  courts,  detention  staff,  and  probation  and  parole 
officials  are  often  coercive  in  nature.  Building  on  the 
reciprocal  effects  and  negative  reinforcement  described  in 
Patterson's  model,  it  followed  that  an  intervention  which 
broke  the  "coercive  cycle"  might  be  effective  in  enhancing 
pro-social  behavior  and  reducing  offending. 

We  were  also  impressed  by  the  success  of  the  Multisystemic 
Therapy  Model  of  Henggeler  and  Borduin  (1990)  and  set 
out  to  fashion  our  intervention  after  their  model. 

Specifically,  we  sought  to  design  a family-based  therapeutic 
intervention  that  would  integrate  affective,  cognitive,  and 
social  interventions,  be  community-based,  incorporate  case 
advocacy,  and  involve  youths  and  families  in  treatment 
planning  and  implementation. 

Boundary  Spanning  and  Collaboration 

In  light  of  the  multiple  systems  in  which  juvenile  offending 
youth  are  involved,  and  guided  by  Child  and  Adolescent 
Service  System  Program  (CASSP;  Stroul  & Friedman, 

1986)  principles,  we  framed  program  development  in  terms 
of  the  Jericho  Principle  (Melton,  1989).  The  Jericho 
Principle  is  a metaphor,  suggesting  that  walls  should  come 
tumbling  down  between  disciplines,  or  sectors,  of  the  child 
and  family  service  system. 

Collaboration  with  Families 

Although  many  families,  for  both  obvious  and  subtle 
reasons,  are  unable  to  provide  consistent  support  for  their 
children,  we  place  a strong  emphasis  on  collaboration  with 
parents  and  families  when  possible.  Families  are  contacted 
while  youths  are  in  detention.  Meetings,  for  assessment, 
treatment  planning  and  counseling,  begin  at  that  point. 
Families  are  involved  in  developing  a structured  transition 
back  to  the  community,  and  in  fine-tuning  services  to  meet 
their  specific  strengths  and  needs. 
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Collaboration  with  the  Juvenile  Justice  System 

The  Prime  Time  Project  is  based  out  of  the  King  County 
Department  of  Youth  Services  (DYS)  and  a nearby 
community  health  clinic.  Therapist/case  managers  of  the 
Prime  Time  Project  make  contact  with  youth  and  families 
while  the  youth  are  detained  and  then  follow  them  into  the 
community  upon  release.  Through  our  physical  placement 
within  juvenile  detention,  we  have  watched  and  perceived 
some  walls  to  have  buckled,  although  not  totally  collapsed, 
between  the  juvenile  justice  and  mental  health  system. 

Judges 

Close  collaboration  with  judges  has  afforded  us  a high 
degree  of  involvement  injudicial  disposition  planning. 
Juvenile  court  judges  refer  youths  to  our  program  and  invite 
our  recommendations  as  to  disposition  planning.  Eager  for 
an  expanded  set  of  options  for  these  troubled  youth,  judges 
have  enthusiastically  welcomed  our  efforts  to  demolish  the 
metaphoric  wall  between  the  justice  and  mental  health 
system. 

Probation  Counselors 

Similarly,  program  staff  have  developed  close  collaborative 
relationships  with  probation  and  parole  officers.  The 
leveling  of  this  wall  has  facilitated  the  design  of  a seamless 
approach  to  behavior  management  across  systems  and 
permitted  more  consistent  monitoring  of  youth's 
participation  and  compliance  in  treatment,  school,  or 
vocational  activities.  Probation  officers  have  welcomed  the 
involvement  of  mental  health  professionals,  leading  to  the 
construction  of  detailed  probation  orders  tied  to  established 
principles  of  behavior  management. 

Collaboration  with  Police 

Working  closely  with  probation  counselors,  we  have  found 
that  responding  to  youths'  non-compliance  with  a warrant 
and  brief  detention  may  serve  as  an  effective  container,  in 
both  the  literal  and  therapeutic  sense.  Recently,  we  have 
been  able  to  forge  a collaborative  alliance  with  local  police 
officials. 

Collaboration  with  Schools 

While  education  represents  an  important  avenue  of  change 
for  youth  in  the  juvenile  justice  system,  school  has  often 
been  the  setting  for  repeated  failure,  both  socially  and  in  the 
classroom.  Among  participants  in  the  Prime  Time  Project, 
few  are  enrolled  in  school  upon  release  from  detention,  and 
almost  all  have  a history  of  multiple  suspensions  or 
expulsions.  Collaboration  with  teachers  and  administrators 
is  vital  to  facilitating  youths'  re-entry  into  the  education 
system. 
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The  Prime  Time  "Day  Time"  Program:  A 
Transitional  Community 

In  light  of  the  degree  to  which  many  youths  are  extruded 
from  their  communities  and  lack  many  of  the  competencies 
necessary  to  social,  academic,  and  vocational  success,  it 
should  be  useful  to  put  in  place  a set  of  structures  to 
facilitate  youths'  transition  towards  broader  community 
involvement.  To  this  end,  the  Prime  Time  Project 
incorporates  a series  of  structured  group  activities.  These 
activities  do  not  take  the  place  of  intensive  family-based 
(when  possible),  eco-systemic  intervention^  Instead,  they 
serve  as  a transitional  community  for  some  youths  and  as  an 
adjunct  for  others. 

Prime  Time  activities  seek  to  provide  additional  structure 
and  external  control  for  youths  who  both  have  too  much 
time  on  their  hands  and  too  little  capacity  for 
impulse-control.  Involvement  in  these  activities  offers  a 
respite  from  the  youth's  involvement  in  coercive 
interactions,  allowing  them  to  practice  alternative, 
pro-social  styles.  Some  youths  benefit  from  a graduated 
return  from  detention  (where  many  report  feeling  quite  safe 
as  a function  of  the  structure  inherent  in  the  setting),  to  the 
community,  where  risks  are  greater  and  external  controls 
limited.  The  structured  daily  activities  may  serve  as  the 
venue  for  the  "in  vivo"  transfer  of  vocational,  academic,  and 
social  skills  in  a setting  located  in  the  community,  while 
featuring  opportunities  for  structured,  pro-social  interaction. 
Finally,  by  designing  activities  that  proceed  through  a series 
of  stages,  youth  are  given  the  opportunity  to  follow  a 
graduated  pathway  of  success  experiences  leading  to 
increasing  community  involvement. 

This  aspect  of  the  intervention  takes  the  form  of  structured 
half-day  and  evening  activities  located  in  a local  community 
center  under  the  umbrella  of  the  community  health  clinic  in 
which  the  program  is  based.  Vocational  training  forms  the 
centerpiece;  youths  spend  two  or  more  days  each  week 
involved  in  hands-on  job  skills  training.  Youths  begin  with 
a set  of  job  readiness  and  basic  skill  building  activities, 
moving  through  a series  of  stages  to  employment  in  the 
community.  Youths  also  participate  one  or  two  days  per 
week  in  training  modules  targeting  areas  such  as  social 
skills,  affect  management,  interpersonal  problem  solving, 
and  drug  and  alcohol  abuse. 

One  evening  per  week,  youths  and  a parent  (or  involved 
adult)  attend  a group  meeting.  Participants  and  program 
staff  share  a meal  together  before  breaking  up  into  separate 
parent  and  youth  groups.  After  youths  and  adults  break  into 
groups,  youths  engage  in  a series  of  activities  designed  to 
afford  an  opportunity  to  practice  pro-social  interactions 
while  working  on  tasks  centered  around  identity 
development.  In  an  adjoining  room,  parents  receive  training 
in  communication  and  behavior  management. 
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Currently,  a pilot  study  with  25  youths  is  being  conducted. 
This  study  is  designed  to  (a)  give  information  about  the 
population  being  served,  (b)  explore  the  effectiveness  of  the 
enrollment  and  data  collection  procedures  that  have  been 
implemented,  and  (c)  give  a preliminary  indication  of  the 
effectiveness  of  the  intervention  in  reaching  desired 
outcomes.  This  pilot  study  will  also  guide  the  preparation  of 
a proposal  for  a larger,  outcome-based  evaluation. 
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Introduction 


Riverside  County's  interagency  program  combines  mental 
health,  educational,  and  probation  services  in  an  effort  to 
reduce  juvenile  recidivism.  The  program  provides 
interagency  services  at  three  therapeutic  high  schools  (12th 
Street,  Perris,  and  Indio  Community  Schools)  and  one 
juvenile  facility  (Van  Horn  Youth  Center)  with  the  goal  of 
keeping  minors  at  home,  and  thereby  reducing  recidivism 
by  keeping  them  out  of  restrictive  facilities  (e.g.,  group 
homes,  juvenile  hall,  California  Youth  Authority). 

Riverside  County's  interagency  services  began  in  FY 
1989-90  under  Assembly  Bill  377  (AB  377)  as  part  of  the 
California  System  of  Care  (SOC)  model  first  developed  in 
Ventura  County  in  FY  1984-85.  After  six  years  of  AB  377 
implementation  (FY  1989-90  through  FY  1994-95), 
Riverside  County  has  served  a total  of  1,507  juvenile  wards 
at  the  interagency  community  schools  and  juvenile  facility. 
The  sites  currently  are  serving  about  300  juvenile  wards  in 
FY  1995-96. 

As  a requirement  of  AB  377,  the  Riverside  County  Mental 
Health  Department  annually  evaluates  the  effectiveness  of 
interagency  services  in  reducing  recidivism  among  juvenile 
court  wards.  Previous  evaluation  found  that  the  number  of 
felony  and  misdemeanor  offenses  significantly  decreased 
for  wards  one  year  after  AB  377  services  compared  to  their 
criminal  activity  the  year  before  such  services  (within 
program  outcomes).  The  number  of  felony  offenses  also 
declined  compared  with  comparison  group  samples 
(between  group  samples). 

Riverside  County's  interagency  program  for  juvenile 
offenders  is  representative  of  the  System  of  Care  model 
being  implemented  in  19  other  California  counties.  The  goal 
of  the  state's  Department  of  Mental  Health  is  to  establish 
System  of  Care  programs  in  all  56  California  counties  and 
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thereby  replicate  statewide  the  positive  recidivism  findings 
that  have  been  reported  in  Riverside  County. 

After  six  years  of  interagency  cooperation  between  the 
County  Office  of  Education,  the  County  Department  of 
Probation,  and  the  County  Department  of  Mental  Health, 
Riverside  County  reports  two  key  outcomes  with 
implications  for  reducing  juvenile  crime  in  California's 
counties: 

• The  total  number  of  felony  and  misdemeanor  offenses 
significantly  decreases  for  Juveniles  one  year  after 
attending  an  interagency  school/facility  compared  to 
their  criminal  activity  the  year  before  being  assigned 
to  program  (within  program  outcomes). 

• Juveniles  are  rearrested  less  often  for  felony  and 
misdemeanor  offenses  one  year  after  attending  an 
interagency  school/  facility  compared  with  youths 
who  attend  non-therapeutic,  non-interagency  sites 
(between  program  outcomes). 

Method 

A pre-treatment  and  post-treatment  design  was  used  to 
evaluate  the  effectiveness  of  interagency  programs  in 
reducing  juvenile  recidivism.  Juvenile  recidivism  was 
measured  one  year  before  receiving  AB  377  services  and 
one  year  after  receiving  such  services.  Four  studies  (FY 
1990-91,  FY  1991-92,  FY  1992-93,  and  FY  1993-94)  were 
conducted  that  employed  the  one  year  pre-treatment  and  one 
year  post-treatment  design.  A fifth  study  was  conducted  (FY 
1994-95)  that  employed  a tworyear  pre-treatment  and 
two-year  post-treatment  design.  Comparison  groups  were 
used  for  the  1990-91,  1992-93,  and  1993-94  studies  to 
establish  a non-treatment  recidivism  baseline. 

The  Riverside  County  Office  of  Education,  the  County 
Department  of  Probation,  and  the  County  Department  of 
Mental  Health  provided  collaborative  educational, 
probational,  and  therapeutic  services  to  wards  at  the 
interagency  high  schools  and  juvenile  facility.  Wards  at  the 
comparison  sites  received  joint  educational  and  probational 
services,  but  did  not  receive  mental  health  services. 

Sites 

Samples  were  selected  from  four  interagency  sites  (Van 
Horn  Youth  Center,  12th  Street  Community  School,  Perris 
Community  School,  and  Indio  Community  School). 

Samples  were  also  selected  from  two  pre-AB  377  sites  (Van 
Horn  and  Perris)  and  from  a non-AB  377  community  school 
(Jurupa)  to  serve  as  comparison  groups.  Juvenile  wards  who 
were  discharged  from  the  four  AB  377  sites  and  who  had  at 
least  two  weeks  of  interagency  services  were  randomly 
selected  for  inclusion  in  the  studies.  For  the  comparison 
group  samples,  juvenile  wards  who  were  discharged  from 
Jurupa  or  a pre-AB  377  program  and  had  attended  that 
school  for  at  least  two  weeks  were  randomly  selected  for  the 
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school  for  at  least  two  weeks  were  randomly  selected  for  the 
studies.  Altogether,  239  wards  receiving  AB  377  services 
were  randomly  selected  for  the  five  recidivism  studies;  71 
wards  were  selected  from  the  comparison  groups. 

Arrest  and  disposition  records  were  examined  for  juvenile 
wards  in  the  four  interagency  samples  and  the  comparison 
group  samples.  These  records  were  collected  from  official 
court  and  probation  department  documents.  The  number  of 
sustained  offenses  by  wards  (i.e.,  felonies,  misdemeanors, 
and  probation  violations)  before  entry  into  an  AB  377 
interagency  program  was  compared  with  the  number  of 
sustained  offenses  after  their  exit  from  the  program.  The 
same  information  was  collected  for  the  pre-AB  377  and 
non-AB  377  comparison  groups. 

Analysis 

Both  inferential  and  descriptive  statistical  were  used  to 
analyze  recidivism  outcomes.  Paired  t-tests  and  t-tests  for 
independent  samples  were  used  to  measure  recidivism 
before  and  after  interagency  intervention.  The  data  is 
currently  also  being  reanalyzed  using  a 2 x 2 (Condition  x 
Time)  repeated  measures  analysis  of  variance  (ANOVA).  In 
addition,  the  effect  sizes  (also  known  as  program  or  clinical 
effectiveness)  corresponding  to  the  inferential  statistics  were 
also  reported.  Two  types  of  significance  testing  were  thus 
reported:  statistical  significance  and  clinical  significance 
(effect  size). 

An  effect  size  is  a statistical  measure  of  program 
effectiveness  that  compares  the  degree  of  a program's 
effectiveness  against  a baseline  or  comparison  group.  A 
program  can  have  a result  that  is  simultaneously  statistically 
non-significant  and  clinically  significant.  One  may  infer 
statistical  significance  from  a clinically  significant  outcome; 
that  is,  a clinically  significant  outcome  denotes  a 
statistically  significant  outcome  (Cohen,  1988;  Thompson, 
1993). 

Results 

Overall,  findings  suggest  that  AB  377  has  been  effective  in 
reducing  the  number  of  felony  offenses  committed  by 
wards.  The  number  of  felony  offenses  committed  by  wards 
at  Van  Horn  and  the  three  community  schools  significantly 
declined  one  year  after  AB  377  services  compared  to  felony 
offenses  committed  one  year  before  such  services 

(t  = 5.40,  df  = 121,  p < .001).  The  number  of  felony  offenses 
also  declined  compared  to  the  comparison  group  samples. 
When  compared  to  a pre-treatment  comparison  sample,  the 
number  of  felony  offenses  committed  by  wards  significantly 
declined  at  Van  Horn  one  year  after  AB  377  services 
compared  to  felony  offenses  committed  one  year  before 
such  services  (t  = 2.03,  df  = 59,  p < .05;  see  Figure  O.  The 
decline  of  felonies  from  pre-treatment  to  post-treatment 
approached  significance  for  wards  at  the  three  community 
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approached  significance  for  wards  at  the  three  community 
schools  (t  = 1 .72,  df  = 203,  p < .09)  when  compared  with 
the  non-AB  377  samples  (see  Figure  2). 

The  differences  in  felony  offenses  between  the  AB  377  sites 
and  the  comparison  samples  were  all  clinically  significant. 
Based  upon  effect  size  analyses,  44%  of  the  wards  from  the 
community  schools  were  likely  to  commit  a felony  after  AB 
377  services  compared  with  56%  of  minors  from  the 
comparison  groups;  37%  of  the  wards  from  Van  Horn  were 
likely  to  commit  a felony  after  services  compared  with  63% 
of  wards  from  the  comparison  group.  Employing  a measure 
of  effectiveness  (Rosenthal,  1984;  Rosenthal  & Rubin, 

1982),  the  AB  377  interagency  program  was  12%  more 
effective  in  reducing  felonies  at  the  community  schools  than 
were  the  comparison  programs;  it  was  also  26%  more 
effective  at  reducing  felonies  at  Van  Horn  than  the 
comparison  group. 

The  effect  size  difference  for  the  community  school  wards 
relative  to  the  comparison  group  was  .25  standard 
deviations,  felony  recidivism  was  thus  reduced  by  a quarter 
standard  deviation.  The  effect  size  difference  for  the  Van 
Horn  wards  relative  to  the  comparison  group  was  .53 
standard  deviations,  felony  recidivism  was  thus  reduced  by 
more  than  half  a standard  deviation.  Employing  Cohen's 
(1988)  measure  of  effect  size  rubric,  the  community  school 
and  Van  Horn  effect  sizes  represent  small  and  medium 
effect  sizes,  respectively. 

The  therapeutic  interagency  programs  were  therefore 
(clinically)  effective  in  reducing  felony  recidivism.  The 
(clinical)  effectiveness  of  the  AB  377  community  schools 
and  Van  Horn  in  reducing  recidivism  compares  favorably 
with  the  success  rates  (effect  sizes)  reported  in  200 
recidivism  studies  reviewed  by  Lipsey  & Wilson  (1992)  that 
show  that  between  43%  to  45%  of  juvenile  wards  were 
likely  to  reoffend  after  intervention  services  compared  to 
54%  to  57%  of  minors  in  programs  without  intervention 
services. 

The  fifth  study  that  employed  a two-year  post-treatment 
design  showed  a decline  over  time  in  juvenile  recidivism 
(see  Figure  3)  based  on  descriptive  statistics  (inferential 
statistics  have  not  yet  been  conducted).  This  study  also 
showed  that  juveniles  who  previously  reoffended  were 
likely  to  reoffend  within  the  first  three  to  six  months  after 
receiving  interagency  System  of  Care  services  (see  Figure 
4).  Wards  who  did  not  reoffend  one  year  after  services  were 
likely  not  to  reoffend  during  the  second  year  after  services. 

Discussion 

California's  System  of  Care  as  implemented  in  Riverside 
County  is  helping  deter  juveniles  from  committing 
additional  crimes.  Riverside  County's  interagency  program 
for  juvenile  offenders  is  similar  to  a model  that  is  also  in 
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place  in  19  other  California  counties.  San  Mateo,  Santa 
Cruz,  and  Ventura  counties  have  also  reported  that  juvenile 
wards  commit  fewer  felonies  and  misdemeanor  crimes  one 
year  after  their  AB  377  System  of  Care  services  compared 
to  their  criminal  activity  the  year  before  such  services. 

The  goal  of  California's  Department  of  Mental  Health  is  to 
establish  system  of  care  programs  in  all  56  counties  and 
thereby  replicate  positive  recidivism  findings  statewide.  The 
positive  recidivism  outcomes  in  Riverside  County  show  that 
the  interagency  System  of  Care  approach  offers  other 
counties  in  California  an  effective  model  for  reducing 
juvenile  criminal  activity.  John  Ryan,  Riverside  County 
Director  of  Mental  Health  and  past  President  of  the 
California  Mental  Health  Director's  Association,  stresses 
that  "We  have  shown  through  research  the  effectiveness  of 
our  collaborative  work  with  juvenile  offenders.  We  have 
also  shown  that  an  interagency  approach  is  more  effective 
when  working  with  juvenile  offenders  than  either  mental 
health  or  probation  trying  to  work  alone." 
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Introduction 


Reforms  including  integration  of  children's  mental  health 
services  have  emerged  in  a variety  of  regions,  communities, 
and  institutions.  Consequently,  psychologists  and  other 
mental  health  professionals  are  increasingly  interested  in 
working  with  schools  to  provide  better,  integrated  services 
(Carlson,  Tharinger,  Bricklin,  DeMers,  & Paavola,  1996; 
Paavola  et  al.,  1996).  To  support  these  collaborations,  it  is 
important  that  child  services  professionals  understand  how 
schools  relate  to  a network  of  care  for  children's  mental 
health  services. 

This  summary  examines  the  role  of  schools  in  integrated 
service  delivery  using  data  from  a project  in  South  Carolina 
that  provides  services  to  children  with  severe  emotional 
disturbance  (SED).  South  Carolina  presents  an  interesting 
case  of  state  child  service  systems  reform;  begirming  in 
1989,  it  was  one  of  the  first  states  to  develop  Medicaid 
services  utilizing  local  education  agencies  as  medical 
providers.  The  Medicaid  office  and  the  Department  of 
Education  identified  potentially  reimbursable  services, 
including  psychological  services.  In  July,  1995,  the 
Medicaid  agency  was  renamed  the  Department  of  Health 
and  Human  Services,  and  supported  services  for  eligible 
children  with  a disability  through  an  Individual  Education 
Plan  (lEP;  Cantrell,  1996).  These  services  were  provided  in 
the  school  building,  using  a strategy  sometimes  known  as 
"push-in"  integrated  therapy.  This  contrasts  with  a strategy 
of  "pull-out"  clinical  services,  in  which  children  needing 
services  are  removed  from  the  classroom  and  school. 

The  involvement  of  schools  in  a network  of  care  for 
children  with  SED  is  important  for  several  reasons.  Schools 
are  potentially  effective  health  delivery  systems  for  children 
because  they  are  stable,  universal,  and  cost-effective 
(Carlson  et  al.,  1996).  To  some  extent,  public  schools  have 
served  to  integrate  children's  social  services  since  their 
inception  (Fagin,  1992).  Schools  may  work  with  families  to 


ensure  that  their  services  are  appropriate  to  both  the  family 
and  the  school's  administration.  In  exceptional  or  special 
student  education,  an  lEP  that  integrates  services  and 
service  providers  is  essential  to  child  health  and  family 
involvement  in  education  (Duchnowski,  Kutash,  & Knitzer, 
1995).  Along  with  inpatient  or  residential  services  and 
mental  health  counseling,  schools  are  a major  axis  of  care 
and  expense  over  time  for  children  with  emotional  or 
behavioral  problems  (Epstein  & Quirm,  1996). 

Additionally,  children  with  serious  emotional  or  behavioral 
disorders  often  receive  care  from  multiple  service  providers. 
Responsibility  for  care  and  health  outcomes  is  shared,  and 
the  degree  to  which  this  shared  responsibility  is  coordinated 
can  be  improved  through  system  interventions  (England  & 
Cole,  1995;  Friedman,  1996).  Under  these  conditions, 
system  coordination  appears  to  be  critical  (Johnsen, 
Morrissey,  & Calloway,  in  press).  The  degree  of  change  in 
integration  and  coordination  of  child  services  may  be 
measured  using  network  analysis  (Morrissey,  1992).  In 
evaluations  of  service  integration  programs  in  upstate  New 
York  and  western  North  Carolina,  social  network  analysis 
demonstrated  that  service  enhancements  such  as  co-locating 
services  and  expanded  case  management,  created  greater 
systems  integration  (Morrissey,  Johnsen,  & Calloway, 

1994). 

In  the  context  of  a national  mandate  to  establish  systems  of 
care  for  children  (Stroul  & Friedman,  1986),  comprehensive 
care  and  full-service  school  models  have  become  prominent 
(Dryfoos,  1994;  Roberts,  1994).  In  terms  of  actual  service 
use,  research’ suggests  that  schools  already  provide  a large 
proportion  of  mental  health  services  for  children  and 
adolescents.  In  western  North  Carolina,  the  Great  Smoky 
Mountain  Study  (GSMS)  focused  on  representative  9-13 
year  olds  attending  public  schools.  In  the  GSMS,  only  40% 
of  children  with  SED  received  services.  Of  children  with 
SED  who  received  services,  only  half  received  them  in  the 
mental  health  sector  (Bums  et  al.,  1995).  Mental  health 
services  in  the  education  sector  were  reported  for  71 .5%  of 
children  with  SED  (vs.  41.5%  who  reported  receiving 
services  from  community  mental  health  providers).  Earlier 
studies  support  the  importance  of  school-based  mental 
health  services,  identifying  teachers  as  the  professional 
group  that  most  often  serves  children  with  mental  health 
problems  (Cohen  et  al.,  1991;  Offer,  Howard,  Schonert,  & 
Ostrov,  1991). 
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Method 

How  are  schools  involved  in  networks  of  care  for  children 
with  SED?  This  study  examined  organizations  participating 
in  a mental  health  services  demonstration  sponsored  by  the 
federal  Center  for  Mental  Health  Services  (CMHS)  and 
administered  by  a Community  Mental  Health  Center  in  an 
urban  area  of  South  Carolina.  A rural  comparison  region 
served  three  nearby  counties.  A systems-level  assessments 
was  designed  to  measure  differences  in  development  and 
coordination  among  service  organizations  in  the 
demonstration  site  (n  = 61)  and  its  comparison  community 
(n  = 39).  In  the  urban  demonstration  site,  organizations 
serving  children  with  SED  were  coordinated  through  a 
system  of  care  model  and  featured  multi-systemic  therapy, 
an  ecological  treatment  model  (Henggeler  & Bourdin,  1990; 
Henggeler  et  al.,  1994).  Baseline  data  were  collected 
between  December,  1994  and  May,  1995.  The  sample  of 
organizations  in  this  study  was  unique  because  the  sample 
was  carefully  designed  to  include  a range  of  schools  in  the 
region.  The  project  was  funded  for  5 years,  and  services 
began  in  March,  1994. 

This  study  used  an  inter-organizational  network  survey 
based  on  Van  de  Ven  and  Ferry's  "Organizational 
Assessment  Instrument"  which  has  been  tested  for  validity 
and  reliability  (Morrissey,  Johnsen,  & Calloway,  1994;  Van 
de  Ven  & Ferry,  1980).  Researchers  gathered  responses 
from  over  90%  of  1 1 7 services  identified  in  the  two 
counties.  Selected  services  were  those  which  performed  a 
CASSP  function  (Stroul  & Friedman,  1986),  were  identified 
as  a key  agency,  and/or  were  used  by  children  with  SED. 

The  survey  instrument  included  detailed  indicators  of  client 
referral  patterns  and  information  exchange  relationships 
among  child  serving  agencies  in  the  region. 

The  inter-organizational  network  of  referral  and 
communication  among  services  for  children  with  SED  was 
summarized  by  indicators  of  social  network  structure, 
including  connectedness,  density,  and  centralization 
(Wasserman  & Faust,  1994).  Network  analysis  techniques 
provided  the  number  and  density  of  referral  and 
communication  ties  to  and  from  particular  service  providers. 
Analysis  at  the  system  level  was  complemented  by  analysis 
of  schools  in  the  network,  including  a comparison  of  those 
urban  schools  with  and  without  in-school  therapeutic 
intervention. 

Results 

The  network  of  service  providers  was  moderately  integrated 
in  both  the  urban  and  rural  regions,  and  more  centralized  in 
the  rural  region.  The  client  referral  patterns  among  61 
organizations  in  the  urban  service  delivery  network  were 
more  dense  (18%  vs.  13%)  and  less  centralized  (.26  vs.  .43) 
than  the  39-organization  rural  network  (see  Table  1). 

Density  is  a measure  of  the  proportion  of  possible  ties  which 
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Density  is  a measure  of  the  proportion  of  possible  ties  which 
are  present,  expressed  as  a proportion  of  non-zero  cells  in  a 
n-by-n  organizational  matrix.  Similarly,  the  information 
exchange  patterns  were  more  dense  in  the  urban  region  (.23 
vs.  .18)  and  less  centralized  than  those  found  in  the  rural 
area.  This  could  have  been  due  to  a smaller  student  and 
client  population  in  the  rural  network,  requiring  fewer 
relationships  between  services  and  allowing  greater 
centralization  of  services.  These  findings  also  may  have 
been  related  to  greater  integration  in  the  urban  region  as  a 
result  of  the  early  effects  of  the  CMHS  project. 

Table  2 reports  a Z-test  for  the  difference  between  two 
proportions  to  compare  the  density  of  system  ties  and  the 
connectivity  of  school  ties  for  rural  and  urban  systems.  The 
urban  density  measures  for  referrals  and  information 
exchanges  were  significantly  higher  than  rural  measures  (p 
<.01). 

The  "connectedness"  of  schools  (the  proportion  of  possible 
ties  to  or  from  schools  to  other  system  respondents)  refers  to 
the  network  density  of  relations  involving  particular  schools 
but  excluding  the  school  district  office. 

Although  the  service  system  as  a whole  was  more  dense, 
urban  region  schools  show  a significantly  lower  degree  of 
connectedness  in  terms  of  the  proportion  of  services  to 
which  they  send  clients  (Z  = 2.4).  In  general,  there  were  no 
significant  differences  in  connectedness;  schools  exchanged 
clients  and  information  with  relatively  few  other  services 
(one  in  ten,  or  less).  The  low  connectedness  measures  were 
partially  due  to  the  fact  that  the  school  district  office,  which 
was  central  and  highly  connected  with  other  services,  was 
excluded  from  the  analysis  of  school  ties. 

In  order  to  examine  the  potential  benefit  of  the 
demonstration.  Table  3 compares  urban  schools  with  and 
without  school-based  counselors  (SBCs).  Proportionately, 
the  urban  system  schools  with  SBCs  sent  clients  to  more 
than  twice  as  many  service  providers  as  the  schools  without 
SBCs  (Z  = 2.66).  Likewise,  the  schools  with  SBCs  sent 
communications,  proportionately,  to  almost  twice  as  many 
other  providers.  This  suggests  that  schools  with  SBCs  may 
actively  initiate  relations  with  other  services  to  a greater 
extent  than  those  without  SBCs.  Most  of  the  schools  in  the 
sample  were  elementary  level,  suggesting  that  urban  region 
SBCs  were  most  active  in  advocating  for  services  for 
younger  children  with  SED. 

Discussion 

Schools  are  important  service  providers  for  children  with 
SED,  but  they  are  not  always  highly  integrated  within  a 
larger  system  of  child  mental  health  services.  This  summary 
described  schools  in  two  districts  that  were  somewhat 
isolated  from  other  providers  in  the  local  service  system. 
School-based  mental  health  counselors,  however,  played  an 
important  role  in  bringing  schools  into  the  system  of  care, 
primarily  through  increased  client  referrals.  The  presence  of 
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primarily  through  increased  client  referrals.  The  presence  of 
school-based  mental  health  counselors  was  associated  with 
more  than  twice  as  many  client  referrals  and  almost  twice  as 
many  information  exchanges  from  schools.  Although  the 
survey  was  conducted  at  an  early  point  in  a demonstration 
project,  it  suggests  that  school-based  mental  health 
counselors  may  increase  the  integration  of  schools  with  the 
larger  system  of  care,  building  bridges  of  referral  and 
information  to  better  serve  the  needs  of  children. 

Since  the  results  reflect  less  than  a year  of  intervention  in 
one  state,  and  since  the  comparisons  were  across  services 
rather  than  over  time,  the  full  impact  of  school-based 
counselors  cannot  be  fully  assessed  using  these  data.  In 
addition,  these  results  are  limited  to  a single  demonstration 
in  a particular  area  and  context.  Future  research  should 
examine  the  role  of  schools  in  different  regional  contexts, 
and  ensure  that  schools  are  included  as  respondents  in 
surveys  regarding  the  provision  of  mental  health  services  for 
children.  Families  and  counselors  should  be  partners  in  both 
health  and  education,  meeting  needs  of  children  in  their 
most  familiar  environments.  Bringing  services  into  schools, 
in  addition  to  referring  children  out  to  services,  may 
improve  both  health  and  education,  in  addition  to  supporting 
the  CASSP  goal  of  providing  services  in  the  least  restrictive 
environment. 

As  interventions  are  implemented,  social  network  analysis 
can  be  used  to  measure  the  impact  of  the  integration  process 
and  to  evaluate  the  degree  to  which  new  service  models  help 
schools  cooperate  within  a system  of  care.  In  addition,  there 
is  some  evidence  that  school  level  (e.g.,  elementaiy  vs.  high 
school)  is  related  to  the  degree  of  service  integration,  so 
developmental  issues  should  be  considered.  Relationships 
between  schools  and  other  SED  service  providers  may  be 
identified  and  promoted  when  they  serve  the  interests  of 
children. 
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Introduction 


Each  year  many  children  and  their  families  present  at 
emergency  service  settings  seeking  help  for  psychiatric 
crises.  Little  is  known  about  the  treatment  that  these 
children  receive  or  the  effectiveness  of  the  treatment 
provided.  If  a child  is  assessed  as  not  in  need  of  immediate 
inpatient  care,  one  option  is  the  provision  of  intensive 
in-home  services.  These  interventions,  often  modeled  on  the 
Homebuilders  program  developed  for  a child  welfare 
population  (Fraser,  Pecora,  & Haapala,  1991),  have  not  been 
systematically  evaluated  to  assess  their  outcomes  when  used 
for  a mental  health  population.  The  purpose  of  this  study 
was  to  assess  the  combined  child,  family,  and  system 
outcomes  of  three  in-home  psychiatric  emergency  programs 
all  located  in  the  same  urban  community. 

In  1987,  the  New  York  State  Office  of  Mental  Health 
established  Home-Based  Crisis  Intervention  (HBCI)  as  an 
intensive  in-home  service  option  for  families  with  a child  in 
psychiatric  crisis.  HBCI  is  modeled  on  the  Homebuilders 
program  developed  in  Tacoma,  Washington  (Kenny, 
Madsen,  Flemming,  & Haapala,  1977).  This  program 
provides  short-term  intensive  services  to  families  with 
children  who  are  at  risk  of  out-of-home  placement. 
Counselors  work  with  two  families  at  any  time  with  a goal 
of  preventing  psychiatric  hospitalization  or  out-of-home 
placement.  The  intervention  focuses  on  family  strengths  and 
needs  using  a multifaceted  approach  including  skills 
building,  counseling,  and  the  provision  of  concrete  services. 

A research  demonstration  grant  (Evans,  1992)  from  the 
National  Institute  of  Mental  Health  (1R18MH50357)  and 
the  Center  for  Mental  Health  Services  (5HD5SM50357) 
allowed  us  to  develop  an  enhanced  HBCI  program 
(HBCI+),  based  in  part  on  lessons  learned  in  prior  research 
(Evans,  et  ah,  1994).  The  study  site  is  located  in  the  Bronx, 
New  York,  a densely  populated  borough  of  New  York  City 
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characterized  by  cultural  diversity  and  intense  poverty.  In 
order  to  answer  questions  about  the  effectiveness  of  these 
two  models,  we  further  modified  a generic  case 
management  program  to  create  Crisis  Case  Management 
(CCM),  an  intensive,  short-term  intervention  whose  purpose 
was  to  do  rapid  assessment  of  child  and  family  needs  and  to 
link  to  needed  services.  The  development  of  CCM  was  to 
answer  the  question  about  whether  a linkage  and  advocacy 
model  of  in-home  services  results  in  the  same  outcomes  as 
in-home  programs  focused  on  skill  building  and  treatment. 

Details  on  all  three  interventions,  which  are  provided 
in-home  for  4 to  6 weeks,  are  available  elsewhere  (Evans, 
Boothroyd,  & Armstrong,  1 996).  In  this  presentation,  we 
will  concentrate  on  aggregate  data  across  the  three  program 
types. 

Method 

Children  were  referred  to  the  study  following  evaluation  by 
a child  and  adolescent  team  at  two  psychiatric  emergency 
rooms.  Children  referred  for  consideration  were  assessed  as 
being  at  risk  of  either  hospitalization  or  out-of-home 
placement  without  immediate  intensive  intervention.  If  a 
child  met  the  study  criteria  and  the  family  and  child  were 
willing  to  participate,  the  family  was  randomly  assigned  to 
one  of  the  three  interventions.  Criteria  for  inclusion  were 
that  the  child  be  (a)  between  the  ages  of  5 and  17;  (b)  living 
in  the  Bronx  with  a natural,  foster  or  adoptive  family;  and 
(c)  experiencing  a psychiatric  crisis  requiring  immediate 
intensive  intervention,  hospitalization,  or  placement  in 
another  restrictive  setting.  Moreover,  the  child's  family  had 
to  be  willing  to  receive  services,  and  the  environment  in 
which  the  child  lived  had  to  be  safe.  Data  on  multiple 
aspects  of  child  and  family  status  and  functioning  were 
collected  on  admission  to  the  study,  at  discharge  from  the 
intervention  (4&endash;6  weeks),  and  six  months  following 
discharge.  Data  reported  in  this  manuscript  are  preliminary 
and  are  based  on  the  first  221  children  admitted  to  the  study. 
Other  presentations  (Evans,  Boothroyd,  & Kuppinger,  1997; 
Boothroyd,  Kuppinger,  & Evans,  1 997)  focus  on  additional 
aspects  of  this  study. 

Results 

To  understand  the  characteristics  of  children  referred  to  the 
study  in  the  context  of  all  children  presenting  at  the 
emergency  rooms,  clinicians  were  asked  to  assess  the 
overall  dangerousness  of  each  child's  condition.  The 
assessment  device  that  was  utilized  was  the  Child  and 
Adolescent  Mental  Health  Assessment  of  Imminent  Danger 
(Gutterman  & Levine,  1 992).  On  a five-point  scale,  ranging 
from  none  (0)  to  severe  (4),  children  referred  for 
hospitalization  (N  = 208)  had  a mean  score  of  2.64;  those 
referred  for  in-home  services  (N  = 1 79)  had  a mean  score  of 
1 .85;  and  those  referred  to  other  community -based  services 
(N  = 620)  had  a mean  score  of  1 .14.  All  between-group 
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(N  = 620)  had  a mean  score  of  1.14.  All  between-group 
differences  are  statistically  significant  at  p = .05.  There  was 
also  a perfect  order  effect  between  the  seriousness  of  a 
child's  condition,  the  child's  assessed  ability  to  participate  in 
services,  and  the  ability  of  the  caretaker  to  provide  a 
supportive  environment  and  participate  in  services  with  the 
treatment  environment  selected  by  the  clinicians  (i.e., 
children  identified  as  having  the  most  severe  clinical 
statuses,  who  were  least  able  to  participate  in  their  own  care 
and  whose  caregivers  were  least  able  to  provide  care,  were 
more  likely  to  be  hospitalized).  Children  at  the  opposite 
extremes  on  these  variables,  however,  were  most  likely  to 
be  referred  for  outpatient  services.  Children  who  scored  as 
intermediate  on  the  risk  variables  were  those  most  often 
referred  to  the  in-home  intervention. 

Additional  characteristics  of  the  22 1 children  served  by  the 
project  in  the  three  intensive,  in-home  interventions  appear 
in  Table  1.  Particularly  remarkable  is  the  proportion  of 
children  with  clinical  diagnoses  of  psychotic  disorders  and 
the  low  incomes  of  their  families,  most  of  which  are  below 
the  poverty  line  (see  Evans,  Boothroyd  & Kuppinger,  1997). 

One  of  the  primary  proximal  outcomes  of  interest  in  this 
study  is  self-esteem.  The  Piers-Harris  Children's 
Self-Concept  Scale  (Piers,  1 984)  was  used  to  measure  this 
variable.  Based  on  the  first  152  children  who  completed  the 
three  intensive  interventions,  the  data  show  that  the  mean 
score  at  intake  was  50.5,  which  was  below  that  of  a 
normative  group  of  children  with  emotional  disturbances. 
The  data  also  show  that  the  mean  score  at  discharge  had 
increased  to  53.7,  which  is  statistically  significant  at  the  .05 
level.  Figure  1 shows  the  subscale  gains  from  entry  to 
discharge.  Children  showed  significant  gains,  ranging  from 
.001  to  .05,  on  all  subscales  except  Popularity.  A smaller 
sample  (N  = 98)  of  children  with  data  from  all  three  data 
collection  points  shows  that  the  gains  made  between  intake 
and  discharge  were  retained  at  six  months. 

Child  satisfaction  with  services  is  assessed  at  discharge 
from  each  of  the  in-home  interventions.  Children  (N  = 158) 
show  high  levels  of  satisfaction  with  nearly  all  aspects  of 
their  care.  The  only  items  with  more  than  20%  of  the 
children  expressing  dissatisfaction  or  a lack  of  clarity  were: 
(a)  knowing  why  the  counselor  or  case  manager  came  to  see 
them  initially;  (b)  being  satisfied  with  the  counselor  or  case 
manager  getting  them  other  services;  and  (c)  being 
terminated  from  service  before  they  were  ready. 

In  regard  to  the  disposition  of  children,  using  the  most 
complete  data  available  at  this  time,  which  includes  the  221 
children  reported  here  and  an  additional  75  children  recently 
admitted,  296  children  and  families  have  been  referred  to 
the  study.  Of  these,  233  completed  the  intervention,  while 
38  were  considered  early  terminations  (i.e.,  they  did  not 
complete  the  entire  intervention).  Twenty-five  of  the 
children  required  hospitalization  at  some  time  during  the 
intervention.  The  most  usual  placement  for  children  at  both 
admission  and  discharge  (88.9%)  was  living  with  their 
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admission  and  discharge  (88.9%)  was  living  with  their 
family  or  other  relatives. 

Discussion 

The  data  presented  reflect  only  a small  sample  of  that 
collected  during  the  research  demonstration,  which  is  still  in 
progress.  Based  on  these  data,  some  questions  have  been 
raised  regarding  whether  in-home  crisis  intervention 
programs  are  serving  as  a hospital  diversion  program  in 
New  York,  as  had  once  been  expected.  Clinicians, 
exercising  their  judgment,  differentiate  among  children  who 
are  referred  for  hospitalization,  for  in-home  services,  or  for 
other  community-based  services.  Despite  the  increased 
capacity  the  grant  resources  have  brought  to  serve  children 
and  families  in  their  homes,  hospitalization  rates  have  not 
changed  significantly  at  the  two  participating  hospitals 
(about  20%).  Rather  than  preventing  hospitalization, 
in-home  services  may  represent  another  community-based 
option  for  children  whose  safety  and  clinical  needs  are  not 
as  acute  as  children  judged  as  requiring  hospitalization.  It 
has  not  been  determined  what  the  hospitalization  rates 
would  have  been  at  these  hospitals  without  the  introduction 
of  these  additional  resources.  Also,  initial  discussions  with 
other  researchers  working  in  this  area  of  study  indicate  that 
the  children  seen  in  emergency  settings  in  the  Bronx  have 
more  serious  problems  than  children  assessed  in  settings 
elsewhere  in  the  country  (Gutterman,  Evans,  Levine, 
Boothroyd,  & Drennan,  1996).  Additional  analyses  are 
needed  to  answer  the  hospital  diversion  question. 

Children  enrolled  in  in-home  services  gained  in  self-esteem, 
and  these  gains  were  maintained  over  the  follow-up  period. 
Also,  most  were  able  to  avoid  hospitalization  during  the 
initial  period  of  service.  Avoidance  of  hospitalization 
assumes  particular  importance  when  considering  the 
following  points:  (a)  that  many  of  these  children  appeared  to 
be  in  crisis  at  the  time  of  their  presentation;  (b)  a significant 
minority  were  diagnosed  with  psychotic  disorders  and/or 
had  been  hospitalized  previously;  and  (c)  the  overall 
dangerousness  of  the  child's  condition,  on  average,  was 
determined  to  be  moderate. 

Since  this  study  is  still  in  progress,  analyses  have  not  yet 
been  performed  to  examine  the  differential  outcomes  that 
may  be  associated  with  the  three  program  models.  There  is 
support,  however,  for  the  assertion  that  children  in 
psychiatric  crisis  and  their  families  can  receive  effective, 
intensive  supportive  care  safely  in  the  community,  and  that 
gains  in  self-esteem  persist  for  at  least  six  months  following 
discharge. 
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Introduction 


Every  parent,  every  service  provider,  and  every  researcher 
working  in  the  field  of  children's  mental  health  shares  the 
common  concern  that  our  service  system  be  able  to  respond 
quickly  and  effectively  to  children  who  are  experiencing  a 
psychiatric  crisis.  Even  as  the  range  of  community-based 
service  options  has  expanded  in  recent  years,  and  even 
though  there  has  been  an  increased  commitment  to  prevent 
unnecessary  hospitalization,  there  is  little  research  available 
to  help  us  understand  which  models  of  crisis  intervention 
are  effective,  for  whom,  and  under  what  circumstances. 

In  1987,  New  York  State  initiated-Home-Based  Crisis 
Intervention  (HBCl),  modeled  on  the  Homebuilders 
program  developed  in  Tacoma,  Washington  (Kinney, 
Madsen,  Fleming,  & Haapala,  1977).  HBCl  provides 
short-term,  intensive,  in-home  services  to  families  whose 
children  are  at  risk  of  out-of-home  placement  because  they 
are  experiencing  a psychiatric  crisis.  Beginning  in  1992 
with  a research  and  demonstration  grant  from  the  National 
Institute  of  Mental  Health  (1R18MHS0357)  and  the  Center 
for  Mental  Health  Services  (SHDSSM50357),  a three-year 
field  experiment  was  conducted  to  evaluate  the  effectiveness 
of  HBCl  and  two  additional  short-term  crisis  intervention 
models.  Enhanced  Home-Based  Crisis  Intervention 
(HBC1+)  and  Crisis  Case  Management  (CCM).  The  purpose 
of  the  study  was  to  evaluate  the  family,  child,  provider,  and 
system  outcomes  of  these  three  interventions.  The  study 
took  place  in  the  Bronx,  New  York,  to  assess  the  impact  of 
these  interventions  with  a population  that  was  ethnically 
diverse  and  under  extreme  economic  stress. 

Each  program  offered  in-home  services  for  four  to  six 
weeks,  with  small  caseloads  ranging  from  two  to  eight 
families  per  worker.  Workers  had  flexible  schedules  and 
were  available  by  beeper,  24  hours  a day,  seven  days  a 
week.  Two  of  the  programs,  HBCl  and  HBC1+,  followed 
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the  Homebuilders  model,  which  employs  a cognitive 
behavioral  approach  for  defusing  crises  and  teaching  skills 
while  linking  families  to  a variety  of  ongoing  services. 
HBCI+  has  the  added  resources  of  in-home  and  out-of-home 
respite  care,  a parent  advocate,  a parent  support  group, 
flexible  service  dollars,  and  additional  training  on  cultural 
issues  as  well  as  on  the  impact  of  violence  on  children.  The 
third  program,  CCM,  is  a short-term  adaptation  of  an 
intensive  case  management  (ICM)  program.  Workers  in 
CCM  carry  four  crisis  cases  and  four  longer-term  ICM 
cases.  The  focus  in  CCM  is  on  assessing  need,  providing 
concrete  services,  and  linking  the  client  to  ongoing  services. 
Families  enrolled  in  CCM  also  have  access  to  respite  care 
services.  More  information  about  the  three  interventions  can 
be  found  elsewhere  (Evans,  Boothroyd,  & Armstrong, 

1996). 

This  summary  describes  the  characteristics  of  the  families 
who  were  enrolled  in  the  study  and  reports  preliminary 
findings  (aggregated  across  all  three  interventions)  on 
family  outcomes  as  measured  by:  (a)  the  Family 
Adaptability  and  Cohesion  Evaluation  Scales  II  (FACES  II; 
Olson,  Fortner,  & Bell,  1982);  (b)  the  Inventory  of  Socially 
Supportive  Behaviors  (ISSB:  Barrera  & Ainlay,  1983);  (c) 
the  Parent/Caregiver  Self-Efficacy  Scale;  and  (d)  the 
Parent/Caregiver  Satisfaction  Survey.  The  latter  two 
measures  were  developed  by  researchers  in  the  Bureau  of 
Evaluation  and  Services  Research  of  the  New  York  State 
Office  of  Mental  Health.  Another  presentation  in  these 
proceedings  (Evans,  Boothroyd,  & Holohean,  1996)  focuses 
on  child  outcomes. 
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Method 

From  November,  1993  through  December,  1995,  all 
children  who  presented  at  two  emergency  rooms  were 
screened  for  eligibility  for  the  study.  To  be  eligible,  children 
had  to  be  between  5 and  17  years  of  age.  Children  also  had 
to  live  in  the  Bronx  in  a natural,  adoptive  or  foster  home 
with  a caregiver  who  was  willing  to  participate,  and  the 
child  had  to  be  able  to  remain  at  home  safely  with  intensive 
services  even  though  they  had  significant  mental  health 
needs.  The  emergency  room  psychiatrists  and  psychiatric 
social  workers  conducted  the  assessments  and  were 
instructed  to  refer  only  children  who  would  likely  be 
hospitalized  in  the  absence  of  crisis  intervention  services. 

All  eligible  children  whose  parents  consented  to  participate 
were  randomly  assigned  to  one  of  the  three  programs.  Data 
were  collected  at  intake,  discharge  from  the  intervention, 
and  six  months  following  discharge.  The  data  reported  here 
are  preliminary.  More  in-depth  analysis  will  be  conducted 
when  data  collection  is  complete  in  the  fall  of  1996.  In  all, 
296  children  were  referred  to  the  study,  233  were 
discharged,  38  withdrew  from  the  study  prior  to  discharge, 
and  25  were  hospitalized  during  the  intervention. 

Descriptive  data  and  satisfaction  data  are  presented  for  the 
approximately  221  families  discharged  as  of  January  1996. 
Data  concerning  changes  from  intake  to  follow-up  are 
presented  for  approximately  115  families  for  whom  six 
month  follow-up  data  were  available. 

Results 

The  families  enrolled  in  this  project  were  representative  of 
the  demographics  of  the  Bronx.  Sixty-three  percent  were 
Hispanic,  and  26%  were  African  American.  Eighty  percent 
of  the  families  had  annual  incomes  of  less  than  $20,000, 
with  53%  below  $10,000.  Only  20%  of  the  primary 
caretakers  were  employed.  Seventy-two  percent  of  the 
caretakers  were  single  parents.  Fifty-five  percent  of  the 
primary  caregivers  had  completed  high  school. 

The  average  age  of  enrolled  children  was  about  1 1 .5  years, 
with  slightly  more  children  (56%)  than  adolescents. 
Disruptive  behavior  (29%)  and  adjustment  disorder  (18%) 
diagnoses  were  the  most  common.  Seventy-one  percent  of 
the  children  had  at  least  one  functional  impairment,  and 
94%  were  judged  to  have  behaviors  that  endangered 
themselves  or  others.  While  the  majority  of  children  were  in 
special  education  (53%),  and  many  had  received  prior 
mental  health  treatment  (42%),  relatively  few  had  been 
previously  hospitalized  (1 1%). 

On  FACES  II,  statistically  significant  gains  were  found  in 
both  cohesion  and  adaptability  from  intake  to  discharge. 

The  six  month  follow-up  assessments  indicate  some  erosion 
in  these  gains,  which  is  statistically  significant  for  the 
cohesion  subscale  (see  Figure  IV 
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cohesion  subscale  fsee  Figure  0. 


A preliminary  analysis  was  conducted  to  determine  if  there 
were  any  child  or  family  characteristics  that  might 
distinguish  between  families  who  experienced  more  positive  j 
versus  more  negative  outcomes  as  measured  by  FACES  II. 
None  of  21  characteristics  examined  (e.g.  age,  gender, 
diagnosis,  family  composition,  maternal  education  level) 
were  found  to  be  significant. 

The  Parent/Caregiver  Self-Efficacy  Scale  assesses  how 
comfortable  parents  feel  with  their  parenting  skills  and 
ability  to  care  for  their  child.  Intake  to  follow-up  gains  in 
caregiver  self-efficacy  are  presented  for  four  of  the  domains 
assessed  (i.e.,  behavioral  management  skills,  dealing  with 
school-related  issues,  ability  to  advocate,  and  providing 
emotional  support).  Of  these,  only  the  gains  in  behavior 
management  and  emotional  support  were  statistically 
significant.  There  was  a statistically  significant  decline  from 
intake  to  follow-up  in  caregivers'  feelings  of  being  able  to 
effectively  meet  the  basic  needs  of  their  children  (see  Fiaure 
2)- 

i 

The  ISSB  indicated  an  extremely  low  level  of  perceived  | 
informal  support  among  all  enrolled  families  (average  score 
= 86  out  of  a possible  score  of  200).  Preliminary  findings 
suggest  that  the  interventions  have  little  impact  here,  and 
that  there  is  actually  a decline  from  discharge  to  follow-up, 
which  is  statistically  significant  (see  Figure  3y 

Each  child  and  primary  caregiver  is  asked  at  discharge  to 
complete  a satisfaction  questionnaire.  Caregivers  were 
particularly  satisfied  with  . the  counselor  or  case  manager's 
advice,  cultural  understanding,  and  handling  of  personal 
issues.  Similarly,  caregivers  were  very  satisfied  with  the 
help  they  received,  their  access  to  the  counselor  or  case 
manager,  and  the  extent  to  which  their  counselor  or  case 
manager  was  well-informed  (see  Fiuure  4'). 

There  was  a little  less  satisfaction  with  the  counselor's  or 
case  manager's  ability  to  link  the  family  with  other  services, 
the  quality  of  skills  teaching,  the  time  it  took  the  worker  to 
contact  them  upon  referral  from  the  emergency  room,  and 
the  overall  frequency  of  contact.  Nevertheless,  even  on 
these  items,  the  percentage  of  caregivers  who  reported  that 
they  were  very  satisfied  never  fell  below  80%. 

Discussion 

The  results  presented  in  this  paper  are  at  first  glance  a small 
subset  of  the  data  collected  during  this  field  study  which  is 
still  in  progress.  While  the  children  are  the  primary  subjects 
of  the  study,  our  examination  includes  an  in-depth  look  at 
the  families  who  care  for  and  support  each  of  these  young 
persons  on  a day-to-day  basis. 

In  this  preliminary  analysis  of  change  on  three 
family/caregiver  measures,  we  have  aggregated  results 
across  all  three  interventions.  Given  the  focus  of  at  least  two 
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across  all  three  interventions.  Given  the  focus  of  at  least  two 
of  the  interventions  on  family  problem  solving  and 
communication,  it  is  not  surprising  to  see  gains  in  both 
dimensions  of  FACES  II  from  intake  to  discharge.  The 
erosion  in  these  gains  from  discharge  to  follow-up  may  be 
related  to  the  fact  that,  while  some  families  were  referred  to 
other  community-based  services  upon  discharge,  few  of 
these  services  were  as  intensive,  and  most  did  not  focus 
specifically  on  maintaining  these  gains. 

As  suggested  by  the  extremely  low  baseline  scores  on  the 
ISSB,  the  families  enrolled  in  these  interventions  are  not 
only  grappling  with  significant  stress,  but  the  majority  are 
doing  so  in  isolation.  None  of  the  interventions  focused 
specifically  on  broadening  family's  networks  of  support,  and 
in  the  six  weeks  from  int^e  to  discharge,  no  significant 
change  was  measured.  A sense  of  loss  following  discharge 
from  an  intensive  intervention  is  one  possible  explanation 
for  the  significant  decline  from  intake  to  follow-up.  In 
future  analyses,  we  will  look  at  this  issue  more  closely  to 
determine  if  there  are  differences  among  the  three 
interventions  or  differences  based  on  the  degree  to  which 
families  participated  in  family  support  and  recreational 
activities.  We  will  also  look  at  whether  there  is  any 
correlation  between  the  perceived  level  of  informal  support 
and  other  outcomes. 

Clearly,  there  are  many  more  analyses  to  be  conducted  to 
explore  the  myriad  questions  embedded  in  this  study.  In 
designing  an  analytic  plan,  we  will  take  a variety  of 
approaches  that  might  include  the  use  of  composite 
measures,  .the  weighting  of  certain  measures  based  on  a 
rationale  associated  with  the  intent  of  the  intervention,  and 
other  strategies.  While  preliminary  findings  support  the 
hypotheses  that  children  in  psychiatric  crisis  can  remain 
safely  at  home  with  intensive  in-home  services,  future 
analyses  will  hopefully  enable  us  to  comment  in  more  detail 
about  which  interventions  work  best  for  which  children  and 
their  families  and  why. 
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Introduction 


The  Bronx  is  a community  plagued  by  chronic  violence.  In 
1994  Bronx  county  was  ranked  first  in  New  York  State 
counties  for  violent  crimes  with  2,316  violent  crimes  per 
100,000  people.  In  1994  the  Bronx  had  the  highest  murder 
rate  of  any  New  York  State  county  (33/100,000) 
representing  20%  of  all  the  state's  murder  arrests.  A story  on 
the  front  page  of  the  Bronx  Daily  News  (Monday, 
September  11,1 995)  indicated  that  only  two  Bronx 
elementary  schools  reported  no  weapons  possession  cases 
during  the  year  and  only  one  school  reported  no  serious 
assaults  on  students  at  school.  Another  story  that  same  day 
reported  the  confession  of  four  Bronx  teenagers  arrested  for 
the  murder  of  a 1 9 year-old  mother  and  a bystander  at  a 
local  playground.  The  motive&emdash;a  dispute  stemming 
from  a softball  game  two  days  earlier.  While  it  is  difficult  to 
specifically  quantify  the  impact  of  community  violence,  the 
behavioral  consequences  of  children  living  in  chronically 
violent  communities  has  been  widely  documented  and 
include  difficulty  concentrating,  memory  impairment, 
increased  anxiety,  aggressiveness,  and  uncaring  behavior 
(Garbarino,  1995). 

During  the  past  four  years  the  NYS  Office  of  Mental  Health 
has  been  conducting  a study  in  the  Bronx,  NY  comparing 
the  effectiveness  of  three  models  of  intensive  in-home, 
emergency  services  as  an  alternative  to  hospitalization  for 
children  and  adolescents  experiencing  psychiatric  crisis. 
Two  hundred  and  ninety-six  eligible  children  were 
randomly  assigned  to  one  of  the  three  models  and  received 
four  to  six  weeks  of  services.  While  the  support  services 
available  to  families  differed  across  the  three  program 
models  (e.g.,  respite  care,  support  groups,  parent  advocacy, 
etc.),  each  model  provided  families  with  intensive  services 
(i.e.,  small  caseload  sizes  ranging  from  2 to  8 cases)  that 
were  available  24  hours  a day,  seven  days  a week  and 
delivered  in  a family's  home. 
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This  summary  (a)  summarizes  the  data  collected  from  some 
of  the  children  and  adolescents  in  this  study  regarding  their 
exposure  to  violence  and  its  effect;  (b)  briefly  describes  the 
training,  technical  assistance,  and  case  supervision 
counselors  received  focused  on  violence  issues;  (c) 
summarizes  selected  strategies  that  workers  use  as  a result 
of  these  efforts  to  address  the  effect  of  violence;  and  (d) 
presents  a case  study  demonstrating  how  violence  was 
interwoven  in  the  life  of  one  child  and  her  family. 

Collection  of  Violence  Data 

Respondents  - Children  surveyed  were  among  those 
children  referred  from  a psychiatric  emergency  room  to 
receive  approximately  four  to  six  weeks  of  intensive 
in-home  services  as  a result  of  a recent  psychiatric 
emergency.  The  children  ranged  in  age  from  5 to  18,  were 
living  at  home  with  their  biological,  foster,  or  adoptive 
families,  and  had  experienced  a psychiatric  crisis  that  would 
have  required  hospitalization  without  the  availability  of 
intensive  in-home  services.  As  of  January  1996,  36  children 
had  been  interviewed  about  their  exposure  to  violence. 

The  ten  counselors  providing  the  intensive  in-home  services 
were  interviewed  in  order  to  retrospectively  obtain 
individual  case  information  for  the  purpose  of  constructing  a 
series  of  case  studies. 

Measures  - During  the  regularly  scheduled  six  month 
follow-up  interview  the  children  who  received  intensive 
in-home  services  were  asked  a series  of  questions  regarding: 
(a)  their  exposure  to  violence,  (b)  how  safe  they  felt  in 
various  settings,  and  (c)  their  knowledge  of  other  children 
carrying  weapons. 

A semi-structured  interview  protocol  was  designed  and  used 
with  counselors  at  the  end  of  the  study.  The  protocol 
focused  on  specific  aspects  of  individual  cases  they  had 
handled  during  the  study  with  particular  emphasis  on 
violence  issues. 

Analysis  - Descriptive  statistics  were  used  to  report 
children's. responses  to  the  questions  on  violence.  Case  study 
methodologies  were  used  to  synthesize  and  summarize  the 
counselor  interviews. 

Findings  from  Interviews  with  Children 

To  what  extent  are  children  exposed  to  violence? 

Figure  1 summarizes  the  responses  of  the  children  to  several 
questions  regarding  their  personal  experience  with  various 
types  of  violence.  As  shown  in  this  figure,  87%  percent  of 
the  children  surveyed  reported  having  witnessed  someone 
being  "beaten  up"  while  34%  reported  that  they  had  been 
beaten  up.  Over  one  quarter  of  the  children  (26%)  indicated 
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beaten  up.  Over  one  quarter  of  the  children  (26%)  indicated 
knowing  someone  who  was  robbed  while  nearly  one  fifth  of 
the  children  (19%)  indicated  having  been  robbed.  Almost 
one  quarter  of  the  children  (23%)  knew  of  someone  who 
had  been  stabbed  while  3%  reported  having  been  stabbed 
themselves.  Although  none  of  the  children  interviewed  had 
been  shot  at,  nearly  a third  of  the  children  (31%)  knew 
someone  who  was  shot  at. 

Do  the  children  have  friends  who  carry  weapons? 

Children  were  asked  if  they  had  friends  who  carried 
weapons.  Approximately  40%  of  the  children  interviewed 
reported  having  friends  who  carry  a knife.  Nearly  a quarter 
of  the  children  (24%)  reported  having  friends  who  carried  a 
gun  while  30%  of  the  children  had  friends  who  carried  some 
other  type  of  weapons.  The  most  popular  among  other 
weapons  were  razor  blades  and  box  cutters. 

Where  and  with  whom  do  children  feel  safe? 

The  children  were  asked  several  questions  about  how  safe 
they  felt  in  various  places  alone  or  with  adults.  These 
responses  are  summarized  in  Figure  2.  As  is  shown  in  this 
figure,  half  of  the  children  (50%)  reported  feeling  unsafe  on 
the  streets  by  themselves.  Forty  percent  felt  unsafe  at  the 
movies,  29%  unsafe  at  school,  and  19%  unsafe  at  home. 
Across  all  settings,  children  reported  feeling  safer  when 
accompanied  by  an  adult;  however,  20%  still  felt  unsafe  on 
the  streets  even  when  they  were  with  an  adult. 

Enhancing  Counselors'  & Case  Managers' 
Abilities  to  Respond  to  Violence 

Three  activities  were  undertaken  in  an  effort  to  enhance 
counselors'  ability  to  respond  to  issues  related  to  violence 
while  working  with  children  and  their  families.  These 
activities  included  the  provision  of  specialized  training, 
technical  assistance,  and  case  supervision  focused  on 
violence  issues. 

Dr.  Marsha  Lewis  from  Edge  Associates  in  Pittsburgh,  PA 
conducted  a two  day  training  program  entitled  "Family 
Systems  Theory  as  a Framework  for  Understanding  Family 
Violence."  This  purpose  of  this  training  was  to  provide 
counselors  and  case  managers  with  a conceptual  framework 
grounded  in  family  systems  theory  to  better  understand  the 
dynamics  of  domestic  violence  and  how  to  implement 
practical  intervention  strategies. 

Dr.  James  Garbarino,  Director  of  the  Family  Life 
Development  Center  at  Cornell  University,  provided 
counselors  with  four  days  of  technical  assistance  and  case 
review.  His  focus  with  the  counselors  was  on  the  negative 
consequences  that  trauma  exerts  on  children  and  the 
development  of  strategies  for  dealing  with  these 
consequences. 
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Ms.  Katherine  Gordy-Levine,  Coordinator  of  the  Enhanced 
Home-Based  Crisis  Intervention  program,  provided  case 
supervision  for  her  staff  on  violence  issues  using  a 
conceptual  framework  she  developed  that  included  three 
intervention  levels:  (a)  providing  support,  which  includes 
meeting  concrete  needs  and  addressing  immediate  safety 
issues  as  well  as  listening  and  affirming;  (b)  enhancing 
cognitive  strategies  through  teaching  skills  (e.g.,  fair 
fighting,  relaxation  techniques)  and  coping  strategies  (e.g., 
developing  planned  responses  to  specific  situations);  and  (c) 
the  use  of  environmental  modifications  (e.g.,  removal  of  a 
family  member,  moving  to  a new  apartment)  or  external 
options  (e.g.,  medication). 

Selected  Strategies  Counselors  and  Case 
Managers  Use  to  Respond  to  Violence 

As  a result  of  the  activities  described  above,  the  counselors 
employed  a variety  of  strategies  in  response  to  the  ongoing 
presence  of  community  and  domestic  violence  as  part  of  the 
intensive  in-home  services  they  provided.  Early  in  the 
intervention  they  conducted  a safety  analysis  of  the  home 
environment  examining  various  aspects  of  the  home  ranging 
from  the  adequacy  of  the  door  locks  to  the  presence  of 
possible  weapons  or  dangerous  implements  in  the  home. 
Flexible  service  dollars  could  be  used  to  purchase  new  locks 
or  make  necessary  safety  repairs.  Plans  were  implemented 
with  families  to  reduce  the  risks  of  other  potential  dangers 
in  the  home.  During  the  intervention  they  promoted 
future-oriented  activities  with  the  child  and  family  such  as 
gardening,  taking  care  of  pets,  and  schooling  to  respond  to 
the  hopelessness  that  typically  accompanies  living  in  a 
community  plagued  with  violence.  Counselors  and  case 
managers  also  promoted  greater  participation  in  face-to-face 
family  activities  such  as  playing  games  to  replace  solitary 
activities  such  as  watching  television.  As  a therapeutic 
response,  they  often  had  children  develop  written  journals  to 
help  them  deal  with  their  exposure  to  trauma. 

The  Story  of  Susan  and  Her  Family 

Susan  (a  pseudonym)  was  a 1 5 year-old  Jamaican  female 
with  no  prior  history  of  mental  health  treatment.  A former 
boyfriend  and  his  friends  broke  into  her  family's  apartment, 
burglarized  it,  and  sexually  assaulted  Susan  and  her  mother. 
In  her  attempt  to  escape  and  get  help,  Susan  jumped  from  a 
second  floor  window,  breaking  both  of  her  legs.  Out  of  fear 
and  anger  stemming  from  this  assault,  Susan's  mother 
refused  to  bring  her  home  from  the  hospital. 

Susan  and  her  family  were  referred  for  intensive  in-home 
services.  First,  the  counselor  met  the  family  prior  to  Susan 
being  released  from  the  hospital  and  helped  facilitate  an 
agreement  between  Susan  and  her  family  that  would  allow 
her  to  come  home.  Flexible  service  dollars  were  used  to 
repair  the  window  and  fix  the  door  locks.  The  counselor 
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taught  the  family  specific  communication  skills  involving 
the  use  of  "I  statements"  and  "feeling  statements"  to 
improve  communication  among  the  family  members  and  to 
reduce  blaming.  The  counselor  also  explained  the  effects  of 
trauma  and  taught  them  relaxation  techniques  to  help  them 
with  the  increased  arousal  they  experienced  in  response  to 
this  event.  The  counselor  helped  the  family  plan  fun 
activities  together  and  referred  them  to  victim  services  and  a 
mental  health  clinic  for  ongoing  support. 

As  a result  of  these  efforts,  Susan  was  able  to  remain  at 
home  with  her  family.  According  to  the  counselor,  the 
number  and  quality  of  the  interactions  between  Susan  and 
her  other  family  members  also  increased  during  the 
intervention.  Finally,  the  family  was  linked  to  support 
services  to  continue  to  help  them  cope  with  the  effects  from 
the  violent  assault. 

Summary  and  Discussion 

In  addition  to  the  specific  psychiatric  needs  of  these 
children,  for  many,  issues  of  violence  have  also  been  a 
persistent  threat.  Given  this  fact,  the  intervention  provided 
to  these  children  and  their  families  has  attempted  to 
systematically  address  violence  issues  within  the  context  of 
the  provision  of  short-term  intensive,  in-home  crisis 
services.  Staff  have  received  specialized  training,  technical 
assistance,  and  case  supervision  throughout  the  study  in  an 
effort  to  incorporate  strategies  that  hopefully  will  begin  to 
ameliorate  some  of  the  negative  consequences  associated 
with  living  in  communities  and  households  where  violence 
is  present. 

The  responses  of  the  children  interviewed  indicate  that 
many  have  witnessed  or  personally  experienced  violent  acts. 
Many  of  the  children  report  feeling  unsafe  when  alone  in 
their  neighborhood,  at  school,  and  most  disturbing,  in  their 
homes.  ^Tiile  the  presence  of  an  adult  helps  promotes 
feelings  of  security  for  many  children,  for  many  others  it 
does  little  to  calm  their  fears.  Their  response&endash;carry 
a weapon  and  provide  one's  own  protection. 

While  the  primary  goal  of  the  services  provided  as  part  of 
this  study  is  to  abate  an  existing  psychiatric  crisis  thereby 
avoiding  hospitalization,  the  need  to  place  emphasis  on 
reducing  the  myriad  of  stressors  that  impact  these  children 
and  families  has  become  increasingly  evident  during  this 
study.  We  cannot  specifically  document  the  degree  to  which 
the  increased  emphasis  placed  on  incorporating  strategies 
for  dealing  with  violence  in  this  study  has  resulted  in 
improved  outcomes  for  children  and  their  families; 
nevertheless,  we  do  know  that  for  Susan,  her  mother,  father, 
and  brother,  the  impact  of  violence  was  central  to  their 
family's  crisis,  and  that  the  counselor's  strategies  allowed 
Susan  to  remain  at  home,  while  helping  the  family  cope 
with  the  lingering  and  dramatic  effects  of  the  violence 
inflicted  upon  them. 
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Introduction 


Much  concern  has  been  expressed  about  the  high  utilization 
rates  of  psychiatric  hospitals,  both  public  and  private,  for 
children  and  adolescents  in  our  nation  and  in  the  State  of 
Tennessee.  These  restrictive  placements  may  not  meet  the 
needs  of  the  child  so  placed,  perhaps  causing  harm  in  some 
cases,  and  are  so  expensive  that  they  deplete  what  are 
already  scarce  fiscal  resources.  The  establishment  of  a 
continuum  of  mental  health  care  that  would  offer  a range  of 
services  has  been  the  goal  of  the  Tennessee  Department  of 
Mental  Health  for  several  years  and  was  a focus  of  the 
Tennessee  Children's  Plan. 

The  Tennessee  Children's  Plan  represents  Tennessee's 
efforts  to  reorganize  the  delivery  of  services  to  children 
through  a coordinated  service  system  for  children,  youth, 
and  families.  The  Children's  Plan  was  phased  in  over  the  1 8 
month  period  between  October,  1991  and  April,  1993. 
Although  state  resources  have  been  channeled  into 
communities  toward  less  restrictive  care,  residential  services 
are  still  heavily  used.  In  Tennessee,  millions  of  state  dollars 
flow  both  into  the  public  and  private  service  sectors  under 
TennCare  (the  Medicaid  waived  managed  care  system  for 
Tennessee)  for  inpatient  psychiatric  care.  In  addition, 
departmental  funds  have  been  used  to  pay  for  residential 
services  not  reimbursed  by  the  TennCare  or  other 
third-party-payor  systems. 

This  ongoing  study  was  designed  to  provide  some  very 
basic  information  for  filling  some  of  the  numerous  gaps  that 
exist  in  the  knowledge  base  about  the  use  of  both  inpatient 
and  residential  mental  health  care  for  children  and 
adolescents  in  state  care.  This  summary  presents  data  from 
three  aspects  of  the  project:  (a)  the  Baseline  Survey,  (b) 
Baseline  - 1995  comparisons,  and  (c)  patterns  of  placements 
leading  to  hospitalization. 
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Method 


Sample 

This  study  focused  on  children  and  youth,  ages  birth 
through  21  years,  who  were  in  the  custody  of  the  State  of 
Tennessee  on  two  points-in-time:  July  31,  1991  (N  = 8,467) 
and  January  31,  1995  (N  = 1 1,131),  representing  "before" 
and  "after"  snapshots  of  the  implementation  of  the 
Tennessee  Children's  Plan.  All  children  in  custody  on  those 
dates  were  included  in  this  study.  The  population,  overall, 
was  55-58%  female  (see  Table  1).  34-37% 
African-American,  and  spread  across  all  age  groups. 
Children  had  been  committed  to  the  custody  of  the 
following  state  agencies:  Youth  Development  (15%); 
Education  (2  - 4%);  Human  Services  (81%);  and  Mental 
Health  and  Mental  Retardation  (<  1 %). 

Sources  of  Data 

Two  existing  state  databases  were  analyzed  for  this  study. 
The  "Modeling  Utilization"  project  uses  existing  state 
databases  and  integrates  information  in  order  to  be  able  to 
link  services  provided  to  children  and  youth  by  multiple 
agencies  over  time.  These  two  databases  were  reviewed  for 
equivalence  of  item  content  and  variables;  the  comparisons 
presented  represent  those  with  items  having  good  match 
between  data  sets. 

Baseline  survey:  1991.  A baseline  survey  was  conducted  by 
the  Tennessee  Commission  on  Children  and  Youth  for  all 
children  in  state  care  on  July  31,  1991.  The  study  was 
designed  to  provide  a baseline  description  of  the  population 
of  children  and  youth  in  state  care  prior  to  the 
implementation  of  the  Tennessee  Children's  Plan. 

1995  Follow-up.  The  Client  Operations  and  Review  System 
(CORS)  database  was  used  for  1995  comparisons.  January 
31,  1995  was  chosen  as  the  point-in-time  comparison  based 
on  completeness  of  the  data  fields  of  interest.  CORS  is 
maintained  for  all  children  served  by  the  Children's  Plan. 

Analyses 

Descriptive  statistics  were  used  to  profile  the  similarities 
and  differences  in  the  characteristics  of  the  youth.  Children 
were  identified  with  mental  health  needs  if  one  or  more  of 
the  following  was  present:  (a)  presence  of  a mental  health 
diagnosis,  (b)  placement  at  any  time  during  custody  in  a 
residential  treatment  center  designated  as  a mental  health 
setting,  or  (c)  placement  at  any  time  in  custody  in  a 
psychiatric  hospital.  To  examine  patterns  of  placement,  the 
six  most  recent  placements  were  arranged  in  sequence  and 
resulting  patterns  were  collapsed  to  form  no  more  than  1 5 
types  of  sequences. 

Results 
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Results 


First,  characteristics  of  the  children  in  custody  in  1991  and 
1995,  respectively,  are  examined.  Next,  the  subpopulation 
of  children  with  identified  mental  health  needs  is  described. 
The  use  of  residential  psychiatric  placements  and  patterns  of 
placement  leading  to  hospitalization  are  then  presented. 

Children  in  Custody  in  Tennessee:  1991  and  1995 

Of  the  total  children  in  custody  in  1991,  almost  half  (47.5%) 
had  some  documented  mental  health  need.  Slightly  more 
than  half  of  the  total  were  males  (see  Table  \).  The  average 
age  of  children  in  state  care  was  12  years,  although  those 
who  were  admitted  to  a psychiatric  hospital  were  slightly 
older  (14  years),  and  those  admitted  by  a Juvenile  Court 
Commitment  Order  (JCCO)  were  older  still  (15.6  years). 
Between  1991  and  1995,  the  total  number  of  children  in 
custody  increased  by  almost  3,000  children.  Only  35%  of 
the  children  in  custody  in  1 995  had  a documented  mental 
health  need.  There  were  only  slight  differences  between 
1991  and  1995  in  the  proportions  who  were  either  male  or 
African-American,  or  of  a given  age  group  for  the  total 
population. 

Most  of  the  increase  in  the  number  of  children  in  custody 
occurred  with  children  who  did  not  have  a documented 
mental  health  need  (from  4,451  to  7,186;  a 61%  increase), 
were  male  (40%  increase),  African-American  (47% 
increase),  young  adolescents  (43%  increase),  and  were 
assigned  to  the  department  of  human  services  (33% 
increase). 

Children  with  Identifled  Mental  Health  Needs:  1991  and 
1995 

Although  the  proportion  of  children  identified  with  mental 
health  needs  decreased  (47.5%  to  35%)  as  the  total  number 
of  children  in  custody  grew,  the  actual  number  of  children 
with  mental  health  needs  remained  relatively  stable  over  this 
period  (from  4,016  to  3,945).  There  was  some  slight 
increase  in  the  proportions  who  were  male  (as  opposed  to 
female),  or  African-American,  and  there  was  a slight 
increase  in  the  proportion  who  were  age  13-15. 

Children  with  mental  health  needs  were  in  state  custody  an 
average  of  761  days  in  1991  and  645  days  in  1995.  They 
most  often  lived  in  foster  care  (44%  in  1991  and  33%  in 
1995;  see  Table  2).  In  1991,  there  were  120  children  in 
public  psychiatric  hospitals  (3%)  and  122  in  these  settings 
in  1995  (also  3%).  In  1991,  there  were  180  children  in 
private  psychiatric  hospitals  (5%),  and  this  number  dropped 
to  49  (1%)  in  1995.  The  placements  for  Level  III  residential 
treatment  centers  (RTC-IIIs),  which  provide  24-hour  care 
for  emotional  and  behavioral  problems,  increased 
dramatically  from  268  (7%)  to  626  (16%)  from  1991  to 
1995. 

Children  in  residential  psychiatric  treatment  had  been  in 
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Children  in  residential  psychiatric  treatment  had  been  in 
their  current  placements,  on  average,  over  6 months  (s^ 
Table  2).  There  was  a decrease  of  34  days  (from  215  to  181) 
in  the  average  days  of  placement  in  a public  psychiatric 
hospital,  but  increases  of  55  days  in  the  length  of  stay  in 
private  psychiatric  hospitals  and  56  days  in  the  length  of 
stay  in  RTC-IIIs.  The  lengths  of  stay  shown  in  Table  2 are 
slightly  misleading  since  they  include  only  the  children  who 
had  not  left  those  settings  and  therefore  include  those  with 
very  long  stays.  For  those  who  had  completed  a residential 
psychiatric  treatment  for  the  1 995  data  only,  it  was  found 
that  the  average  days  in  placement  for  completed  residential 
treatment  was  92  days  for  stays  in  public  psychiatric 
hospitals,  89  days  for  stays  in  private  psychiatric  hospitals, 
and  163  days  for  completed  stays  in  RTC-IIIs.  In  1991,  22% 
of  the  children  currently  in  public  psychiatric  hospitals  had 
been  there  30  days  or  less,  in  1995  the  comparable  figure 
was  1 8%.  The  corresponding  figures  for  private  psychiatric 
hospitals  were  32%  for  1991  and  22%  for  1995.  For  those 
staying  over  one  year  in  public  psychiatric  hospitals,  the 
percentage  was  1 1%  in  1991,  and  for  1995,  it  was  8%;  and 
for  private  psychiatric  hospitals  5%  had  stayed  over  one 
year  in  1991,  and  14%  had  stayed  this  long  in  1995.  The 
actual  maximum  days  for  a current  psychiatric 
hospitalization  in  a public  hospital  were  3,378  days  in  1991 
and  1,076  days  in  1995.  The  maximum  days  for  private 
psychiatric  hospitalizations  were  771  for  1991  and  717  for 
1995.  In  1995,  the  total  days  of  residential  psychiatric  care, 
which  includes  psychiatric  hospitalization  and  RTC  III 
placements,  was  211%  of  that  for  1991,  increasing  from  a 
total  of  97,044  in  1991  to  206,307  in  1995. 

When  compared  to  1991,  children  in  psychiatric  hospitals  in 
1 995  were  more  likely  to  be  African-American  (42%  vs. 
22%),  and  more  likely  to  be  in  the  custody  of  the 
Department  of  Youth  Development  (15%  vs.  8%).  Children 
in  residential  treatment  centers  in  1995  were  more  likely  to 
be  male  (82%  vs.  67%)  and  in  the  custody  of  the 
Department  of  Youth  Development  (24%  vs.  18%)  when 
compared  to  1991 . For  both  years,  the  most  common 
diagnoses  listed  for  children  in  residential  psychiatric 
treatment  were  behavior  problems,  depression,  and 
adjustment  disorders. 

Psychiatric  Hospital  Placements 

For  the  1995  data,  reflecting  only  those  children  in  state 
custody  on  January  31,  1995,  considerably  more 
information  was  available  on  placement  history.  A total  of 
903  children  had  been  placed  in  a psychiatric  hospital 
within  the  current  custody  period  and  the  past  6 placements. 
This  amounts  to  8%  of  all  children  in  custody  and  23%  of 
all  children  with  mental  health  need  who  were  in  custody  on 
January  31,  1995.  Of  those  903  children,  19%  were 
currently  (i.e.,  on  the  target  date)  in  a psychiatric  hospital 
(122  children  in  public  hospitals  and  49  in  private 
hospitals).  A total  of  52%  had  been  in  a psychiatric  hospital 
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multiple  times  while  in  custody:  429  (48%)  had  only  one 
hospitalization,  349  (39%)  had  two  hospitalizations,  10% 
had  3-5  hospitalizations  and  3%  (26  children)  had  6 or  more 
hospitalizations.  For  the  171  children  currently  in 
psychiatric  hospitals  on  the  target  date  in  1995,  this  was  the 
initial  custody  placement  for  26  children  (15%).  The 
placement  experiences  since  entering  state  custody  of  the 
children  currently  in  psychiatric  hospitals  were  varied: 

• 23%  in  a private  psychiatric  hospital  only; 

• 16%  in  a public  psychiatric  hospital  only; 

• 4%  in  both  public  and  private  psychiatric  hospitals; 

• 10%  in  a RTC  and  a psychiatric  hospital; 

• 8%  in  an  assessment  center  and  a psychiatric  hospital; 

• 8%  in  foster  care  and  a psychiatric  hospital; 

• 7%  in  a family  setting  and  a psychiatric  hospital; 

• 5%  in  a Youth  Development  setting  and  a psychiatric 
hospital;  and 

• 2%  in  some  other  setting  and  a psychiatric  hospital. 

Most  referrals  to  a psychiatric  hospital  might  better  be 
called  transfers  since  the  level  of  care  or  restrictiveness  of 
the  setting  did  not  change.  A full  28%  of  those  currently  in 
public  psychiatric  hospitals  had  come  from  another  public 
psychiatric  hospital  program  to  the  current  setting.  For 
current  private  psychiatric  hospitalizations,  this  percentage 
was  27%.  However,  from  the  child's  perspective,  even  these 
transfers  represent  an  actual  change  in  their  living 
environment  and  the  accompanying  stressors. 

Discussion 

Use  of  restrictive,  residential  psychiatric  placements 
increased  from  1991  to  1995.  These  data  indicate  that  the 
Children's  Plan,  while  decreasing  the  use  of  psychiatric 
hospitals,  has  shifted  children  to  longer-stay  residential 
treatment  centers.  Any  savings  that  might  have  been 
incurred  through  decreased  use  of  hospitalization  has  been 
more  than  taken  up  through  this  increased  use  of  RTCs. 
Furthermore,  a significant  proportion  of  the  children  in  state 
custody  experience  multiple  and  long-stay  institutional 
experiences. 

The  description  of  children  in  custody,  those  identified  with 
mental  health  needs,  and  placement  patterns  were  the  focus 
of  this  summary.  These  are  first  steps  needed  to  provide 
background  information  about  the  population  and  use  of 
residential  placements  before  proceeding  with  further  efforts 
toward  modeling  service  utilization  and  patterns  of  care. 
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Introduction 


Historically,  children  have  been  placed  in  out-of-home  care 
in  alarming  numbers.  For  example,  between  October  1985 
and  June,  1988,  102,000  youth  in  the  US  sought  services  in 
shelter  care  programs  (US  General  Accounting  Office, 

1989).  In  1990  alone,  juvenile  correctional  facilities 
reported  98,000  admissions  to  long-term  public  facilities 
(Snyder  & Sickmund,  1 995).  As  would  be  expected,  youth 
who  find  themselves  in  these  settings  have  a variety  of 
service  needs.  This  summary  presents  preliminary  evidence 
for  the  effectiveness  of  a model  of  care  designed  to  provide 
safe  and  effective  services  in  both  short-term  shelter  and 
short-term  staff  secure  detention  programs. 

Initially,  Boys  Town's  short-term  crisis  shelter  programs 
were  designed  to  provide  a safe  and  therapeutic 
environment  for  homeless  and  runaway  youth  in  need  of 
short-term  crisis  services.  Over  time,  the  mission  of  the 
shelter  program  changed  and  evolved,  and  the  program  now 
provides  a range  of  short-term  residential  programs.  These 
programs  serve  youth  with  a variety  of  residential  needs, 
including  homeless/runaway  youth,  troubled  youth  in  need 
of  a short-term  residential  facility,  and  youth  in  need  of  a 
staff  secure  detention  facility. 

Most  recently,  efforts  have  focused  on  developing  strategies 
to  serve  youth  in  need  of  staff  secure  detention  facilities. 
Staff  secure  detention  programs  are  short-term  residential 
settings  for  youth  awaiting  adjudication,  disposition,  or 
placement.  The  term  "staff  secure"  indicates  that  these 
facilities  are  not  locked  or  guarded.  Currently,  Boys  Town 
operates  one  program  designated  as  a short-term,  staff 
secure  detention  facility  in  Brooklyn,  New  York.  This 
facility  serves  only  males  who  are  referred  by  Brooklyn's 
juvenile  justice  system.  This  program  is  designed  to  reflect  a 
guiding  principle  within  juvenile  justice:  Juveniles  should 
be  housed  in  the  least  restrictive  placement  alternative;  this 
is  true  for  both  short-term  and  long-term  placement  (Snyder 
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& Sickmund,  1995). 


Boys  Town  currently  operates  a total  of  eight  shelter 
programs.  In  this  summary,  data  obtained  from  over  2,000 
admissions  to  these  programs  between  January,  1994  and 
July,  1995  are  presented.  These  programs  are  located  in 
Brooklyn,  NY;  Los  Angeles,  CA;  Oveido,  FL;  San  Antonio, 
TX;  New  Orleans,  LA;  Las  Vegas,  NV;  and  Grand  Island, 
NE.  The  eighth  program  in  Washington,  DC  opened  in 
August  of  1995;  there  are  no  data  from  this  site  to  report  at 
this  time. 

Method 

Youth  from  each  of  these  shelters  demonstrate  remarkable 
similarity  in  their  profiles.  Our  research  indicated  that  youth 
are  all  about  the  same  age  (i.e.,  14  years),  and  have  similar 
problems&endash;both  at  the  individual  and  family  level. 

At  intake,  all  youth  were  asked  a series  of  37  questions 
regarding  problems  they  have  experienced.  Of  these,  1 1 
problems  emerged  as  the  most  common  youth  problems 
across  all  seven  sites.  These  youth  problems  were  best 
characterized  as  delinquent  behavior,  school  problems, 
substance  use  issues,  out  of  control  behavior,  and  mental 
health  issues.  A similar  pattern  was  evident  with  family 
stressors,  with  17  stressors  out  of  34  emerging  as  the  most 
common  across  all  seven  sites.  These  stressors  were  best 
characterized  as  substance  use,  criminal  involvement, 
parental  marital  problems,  parental  financial  issues,  and 
parental  discipline  issues.  Youth  also  were  administered  the 
Achenbach  Youth  Self-Report  (YSR;  Achenbach,  1991)  at 
intake.  Youth  from  all  sites  had  similar  YSR  profiles.  There 
was  little  variation  among  these  profiles  on  any  of  the 
subscales  or  broad  band  scales,  with  none  of  the  subscales 
or  broad  band  scales  approaching  the  clinical  range. 

Relative  to  the  other  scales,  the  only  subscale  that  showed  a 
slight  elevation  was  the  Delinquent  subscale.  Although  the 
Delinquent  Behavior  subscale  was  elevated  across  all  sites, 
there  was  no  variation  among  the  sites  on  this  dimension. 

Results 

Indicators  of  Program  Success 

One  indicator  of  the  success  of  these  programs,  both  for 
short-term  residential  and  staff  secure  detention,  was  the 
low  occurrence  of  negative  incidents  as  measured  by  an 
incident  index.  This  index  is  comprised  of  14  negative 
incident  codes.  Examples  of  these  codes  include  physical 
aggression,  out  of  control  behavior,  runaway,  inappropriate 
behavior,  suicidal  gestures,  etc.  The  average  number  of 
negative  incidents  per  youth  per  length  of  stay  ranged  from 
2.42  to  8.73,  computed  by  taking  the  total  number  of 
incidents  per  site  and  dividing  that  number  by  the  total 
number  of  youth  per  site.  Broken  dovra  further,  the  average 
daily  number  of  negative  incidents  ranged  from  .1 5 to  .41 , 
computed  by  dividing  the  average  number  of  negative 
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computed  by  dividing  the  average  number  of  negative 
incidents  per  site  by  the  average  length  of  stay.  The  average 
number  of  negative  incidents  per  youth  per  stay  was 
relatively  low  across  all  sites,  including  the  NY  site  which 
houses  youth  in  need  of  a staff  secure  detention  facility.  The 
background  of  the  NY  site's  youth  might  lead  one  to  expect 
this  site  to  have  far  more  negative  incidents  than  the  other 
sites.  Although  the  NY  site  did  have  the  greatest  number  of 
negative  incidents  (.41  per  day),  this  rate  was  not 
inconsistent  with  the  other  sites. 

Another  indicator  of  program  success  was  the  results  of  the 
satisfaction  survey  completed  by  youth.  At  departure  from 
the  program,  all  youth  were  asked  to  complete  a satisfaction 
survey.  The  survey  is  comprised  of  nine  items  and  asks 
questions  regarding  the  youths'  satisfaction  on  the  following 
dimensions:  staff  fairness;  freedom  to  discuss  problems  with 
staff;  staff  concern  for  youth;  staff  pleasantness;  staffs 
efforts  with  helping  youth  to  get  along  with  others; 
recreation  activities;  counseling;  contact  with  family;  and 
sharing  opinions.  An  average  Overall  score  was  computed 
by  collapsing  the  subscale  scores  across  all  dimensions.  The 
scale  was  based  on  a 7 point  Likert-type  scale,  with  1 = 
completely  dissatisfied  and  7 = completely  satisfied.  Survey 
results  were  consistently  high  across  all  sites,  with  overall 
scores  ranging  from  5.92  to  6.23.  Satisfaction  ratings  across 
all  dimensions  were  high  across  all  sites  indicating  that 
youth  reported  that  they  were  satisfied  with  the  program, 
irrespective  of  the  program's  focus. 

Placement  data  showing  whether  a youth  was  placed  in 
another  out-of-home  setting  or  reunified  with  his/her  family 
■provided  an  additional  indicator  for  program  success.  Data 
was  obtained  for  every  youth  at  departure  regarding  their 
placement  upon  completion  of  the  program.  The 
Restrictiveness  of  Living  Environment  Scales  (ROLES; 
Hawkins,  Almeida,  Fabry,  & Reitz,  1992)  was  used  to 
categorize  these  placements  (e.g.,  group  home,  county 
detention  center,  foster  care,  etc).  The  percentage  of  youth 
reunified  with  their  families  ranged  from  23 . 1 % to  78. 1 %. 

Reunification  appeared  to  be  directly  related  to  the  purpose 
of  the  shelter  site  (i.e.,  whether  the  shelter  functioned  as  a 
short-term  residential  placement  or  not).  For  example,  sites 
which  operated  as  short-term  residential  facilities  had  a 
fairly  high  rate  of  placing  the  youth  with  a parent  or  relative 
at  departure  (FL,  NE,  TX,  & CA).  The  other  sites  had  much 
lower  placement  rates  with  parents  or  relatives,  which 
seemed  to  correspond  to  site-specific  characteristics  which 
impacted  reunification  for  youth  served.  For  example,  the 
Louisiana  site  served  a large  population  of  youth  who  were 
homeless  due  to  abandonment  by  their  parents  or  to 
situations  such  as  incarceration  of  a parent.  Thus, 
reunification  was  not  an  option  for  these  youth.  Similarly, 
the  NY  site  was  used  as  staff  secure  detention  for  many 
youth  on  their  way  to  a juvenile  detention  center.  Although 
the  reunification  percentage  in  NY  was  low  compared  to  the 
other  sites  (34.4%),  it  is  worth  noting  that  this  percentage  is 
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remarkably  high  given  that  these  youth  were  being  detained 
by  the  juvenile  court  awaiting  adjudication. 

Conclusion 

In  summary,  these  data  suggest  that  a model  of  care 
developed  for  crisis  shelter  care  can  be  replicated  in 
geographically  diverse  locations  with  youth  who  have  a 
variety  of  serious  personal  and  family  problems.  These  data 
also  suggest  that  this  model  can  be  successfully 
implemented  in  short-term  residential  and  staff  secure 
detention  programs. 
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Introduction 


With  the  JCAHO's  "Agenda  for  Change"  and  the  advent  of 
managed  care,  outcome  assessments  have  become 
increasingly  important  to  psychiatric  hospitals  (JCAHO, 
1992).  In  response  to  these  new  challenges,  a Severity  of 
Illness  Rating  Scale  (SIRS)  was  incorporated  into  the 
clinical  assessment  process  at  Arthur  Brisbane  Child 
Treatment  Center  to  determine  the  appropriateness  for 
admissions  and  continued  stays.  Although  the  current  field 
is  replete  with  measures  for  assessing  the  clinical  status  of 
adolescents  in  need  of  psychiatric  inpatient  treatment,  the 
validated  measures  were  assessed  as  either  not  easily 
administered  or  not  translating  well  into  our  decision 
making  process  regarding  hospitalization  (Wetzler,  1989; 
Boy-Byrne,  Dagadakis,  Ries  et  al.,  1995).  The  clinical 
leadership  selected  the  SIRS  (source  unknown  at  this  time) 
based  upon  its  ease  of  administration  and  its  face  validity 
with  results  directly  relating  to  the  assessment  for  continued 
inpatient  hospitalization. 

This  study  explores  and  speculates  upon  the  significance  of 
the  Severity  of  Illness  Rating  Scale  (SIRS)  as  a tool  for 
utilization  review  and  program  evaluation.  The  SIRS  was 
first  introduced  March,  1 993  and  soon  became  the 
centerpiece  for  the  hospital's  Utilization  Review  Plan.  Since 
its  implementation,  the  monthly  average  in-house  lengths  of 
stay  at  Arthur  Brisbane  (an  intermediate  to  long  term  state 
psychiatric  hospital)  have  decreased  142  days  from  234 
days  in  March,  1993  to  92  days  in  February,  1996.  This  is 
remarkable  considering  that  our  Hospital  has  not  had  any 
external  managed  care  review  process  to  impact  upon  days 
hospitalized. 

SIRS  scores  have  provided  the  following:  (a)  objective 
criteria  used  to  determine  the  appropriateness  of  admissions 
and  continued  stays;  (b)  clinical  profiles  by  adolescent,  unit, 
facility-wide,  and  attending  psychiatrist;  (c)  clinical 
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outcomes;  (d)  a flagging  system  for  special  treatment 
reviews;  (e)  a vehicle  to  predict  lengths  of  stay;  and  (f)  a 
vehicle  to  validate  both  the  facility's  Mission  and  the 
clinical  program  design. 

Method 

The  sample  in  this  study  consists  of  63  adolescents  (33 
female  and  30  male)  who  were  admitted  and  discharged 
between  August  1,  1994,  and  July  31,  1995.  The  adolescents 
were  between  the  ages  of  1 1 and  1 7 years  old  with  a mean 
age  of  15.2.  This  group  was  47.6%  (n  = 30)  white,  17.5%  (n 
= 1 1)  Hispanic,  33.3%  (n  = 21)  Black,  and  1.6%  (n  = 1) 
Oriental.  Of  the  63  adolescents,  25  were  discharged  to 
home,  10  returned  to  the  juvenile  justice  system,  and  28 
were  discharged  to  residential  placements.  The  average  stay 
of  this  sample  was  91 .2  days,  ranging  from  4 to  286  days. 

The  treating  psychiatrist  assigned  the  adolescents  their  SIRS 
scores  based  upon  the  evaluation  of  the  adolescents'  clinical 
condition  at  the  time  of  admission  and  at  subsequent 
treatment  team  reviews.  For  the  purpose  of  this  study,  focus 
was  placed  on  the  admission  and  discharge  scores.  The 
SIRS  scores  numerically  indicate  the  degree  of  functional 
impairment  in  eight  separate  categories  of  functioning:  (a) 
affective  stability  (AS),  (b)  behavioral  impulsivity  (BI),  (c) 
thought  process  (TP),  (d)  interpersonal  relationships  (IR), 

(e)  problem  solving  skills  (PSS),  (f)  social  support  network 
(SSN),  (g)  danger  to  self  (DS),  (h)  and  danger  to  others 
(DO).  The  scheduled  assignment  of  SIRS  scores  translates 
into  a profile  of  the  adolescents'  monthly  progress  from 
admission  to  discharge.  The  scores  are  assigned  on  a 1 
through  5 Likert-type  Scale,  with  1 representing  healthy 
functional  behavior  and  5 representing  the  highest  degree  of 
dysfunctional  behavior.  Thus,  the  higher  the  SIRS  score  the 
more  serious  the  severity  of  illness.  Scores  of  4 or  5 in  the 
italicized  categories  above  would  theoretically  indicate  a 
need  for  hospitalization. 

The  scale  shares  many  characteristics  of  the  Global 
Assessment  of  Functioning  (GAF;  Goldman,  Skodal,  & 
Lave,  1992).  However,  it  reduces  the  range  of  responses  to 
five  anchored  scores  across  8 separate  dimensions.  Ratings 
by  four  treating  psychiatrists  and  three  covering 
psychiatrists  completed  on  average  two  weeks  apart  for  1 6 
adolescent  yielded  significantly  correlated  interrater 
reliability  (Spearman  r),  with  an  r = .70,  (p  < .01).  The 
psychiatrists  subsequently  attributed  the  differences 
between  ratings  to  clinical  opinion,  change  in  adolescents' 
functioning,  and  limitation  of  information  by  the  covering 
psychiatrist. 

Results 

The  average  admissions  SIRS  scores,  discharge  settings 
(DISC:  1 = juvenile  justice,  2 = home,  3 = residential),  and 
the  adolescents'  age  proved  to  be  reasonably  good  predictors 
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the  adolescents'  age  proved  to  be  reasonably  good  predictors 
of  lengths  of  stay  (LOS).  Using  a stepwise  regression 
procedure,  these  three  variables  account  for  39%  (R2  = .39, 
N = 63)  of  the  variance.  Table  1 presents  a summary  of  the 
strength  of  these  relationships  based  upon  the  stepwise 
procedure  for  predicting  LOS. 

Adolescents  with  higher  SIRS  scores,  who  were  placed 
residentially  and  who  were  younger,  have  longer  stays.  The 
best  equation  for  predicting  LOS  from  this  sample  was: 

LOS  = .86  + 58.33  (Avg.  SIRS)  + 30.94  (DISC)  - 10.67 
(AGE). 

In  regard  to  race,  sex,  or  program  assignment,  no  significant 
relationships  were  observed  between  each  of  these  variables 
and  LOS  which  might  result  in  unintended  variation  (see 
Table  2). 

Though  gender  was  not  significant  in  terms  of  LOS,  females 
on  average  did  take  a little  longer  to  place  than  males.  The 
difference  disappears  with  an  average  of  1 19  days  for 
females  (n  = 3 1)  compared  to  1 1 0 days  for  males  (n  = 22) 
when  you  exclude  the  10  adolescents  returned  to  juvenile 
Justice  placements. 

Conceptually,  the  Hospital's  program  design  consists  of  the 
Cottage  and  the  Main  House  Programs.  The  Cottages 
generally  provide  treatment  for  adolescents  who  externalize 
their  behaviors  and  the  Main  House  for  those  who 
internalize  their  behaviors.  Adolescents  served,  however, 
fall  along  a continuum  between  the  two  points  and  do  not  fit 
easily  into  such  classification.  This  study  examined  whether 
or  not  the  profiles  demonstrated  by  the  SIRS  scores 
validated  this  split. 

Differences  between  the  externalized  and  internalized 
programs  were  not  as  strong  as  initially  anticipated.  We 
expected  the  Main  House  (internalized)  Program  to  have 
greater  disturbance  in  AS  and  DS  and  less  on  DO.  The 
programs,  however,  only  differed  significantly  in  AS.  The 
difference  in  the  other  means,  however,  were  in  the 
expected  direction  (see  Table  3). 

BI,  TP,  PSS,  IR,  and  SSN  were  tested  and  found  not  to 
significantly  vary  between  programs. 

The  data  accommodates  comparisons  for  LOS  and  changes 
in  average  SIRS  scores  among  psychiatrists,  which  allows 
for  assessing  practitioners'  performance.  Direct  comparison 
of  mean  LOS's  for  each  of  the  four  psychiatrists 
demonstrated  a maximum  difference  of  46  days  with  the 
low  mean  at  77  days  and  the  high  mean  at  123  days. 
However,  such  a comparison  needs  to  adjust  for  admissions' 
SIRS  scores,  the  discharge  settings,  and  age  of  the 
adolescents  since  these  were  found  to  relate  to  LOS. 
Controlling  for  these  variables  through  an  analysis  of 
covariance,  only  a 17  day  difference  among  the  adjusted 
means  was  found  with  none  being  significantly  different 
from  the  others  (see  Table  4). 
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from  the  others  (see  Table  4V 

Interestingly,  the  longer  the  stays,  the  greater  the  change  in 
the  average  SIRS  scores.  The  adjusted  LOS  means  and 
average  changes  in  SIRS  scores  provided  a meaningful 
appraisal  of  the  performance  of  each  adolescent  as 
compared  to  his  or  her  peers. 

The  average  clinical  profile  upon  admission  described  an 
adolescent  with  poor  control  of  impulses  (BI  = 3.6),  poor 
social  support  (SSN  = 3.3),  poor  problem  solving  skills 
(PSS  = 3.5),  and  who  was  a danger  to  self  (DS  = 3.6)  or 
others  (DO  = 3.2).  This  profile  supports  the  Mission  of  the 
Center;  upon  admission  adolescents  received  a score  of  4 or 
higher  in  at  least  one  of  the  categories  which  was  consistent 
with  a need  for  continued  inpatient  treatment. 

As  would  be  expected,  almost  all  adolescents  improved  in 
their  average  SIRS  scores  from  the  time  of  admission  (avg. 
3.3)  to  their  final  rating  prior  to  discharge  (avg.  2.8).  The 
greatest  average  SIRS  score  improvements  are  noted  in  the 
categories  of  BI  (.7),  DS  (.7) , DO  (.6),  AS  (.5),  and  PSS 
(.5).  All  but  SSN  were  statistically  significant.  Table  5 
summarizes  the  average  reduction.. 

This  profile  provides  some  evidence  of  the  dimensions  of 
adolescent  functioning  in  which  our  hospital  makes  the 
most  favorable  clinical  impact.  The  changes  demonstrated 
in  the  average  SIRS  scores  represent  a verification  of  the 
anticipated  outcomes  of  target  goals  for  patient  treatment 
planning,  especially  the  efficaciousness  of  managing 
dangerous  behaviors. 

Discussion 

The  admission  average  SIRS  score,  discharge  setting,  and 
the  adolescent's  age  proved  to  be  the  best  organizational 
predictors  for  LOS.  The  SIRS  scores  supported  the 
appropriateness  of  admissions,  and  to  a lesser  extent, 
program  assignments.  In  addition,  it  provided  a fair 
approach  to  assessing  psychiatrists'  performance.  The  SIRS 
scores  also  reported  improvement  from  the  point  of 
admission  to  discharge  in  7 out  of  8 categories  of 
functioning,  with  little  improvement  in  SSN.  The  profile  of 
the  adolescents  shows  that  the  functions  that  are  major 
determinants  for  hospitalization  (i.e.,  BI,  DS,  and  DO) 
improved  the  most.  According  to  this  measure,  the 
outcomes  of  treatment  indicated  that  Arthur  Brisbane 
Hospital's  treatment  program  design  accomplished  its 
mission 

SIRS  has  been  an  effective  tool  in  managing  the  care  of 
adolescents  at  Arthur  Brisbane.  Further  study  is  indicated  to 
determine  its  applicability  in  acute  psychiatric  settings. 
Additional  work  is  also  needed  to  strengthen  its  reliability, 
which  in  turn,  may  further  improve  its  predictive  powers. 
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Introduction 

Public  managed  care  practices  are  in  some  stage  of 
development  in  each  state.  The  implications  of  these 
policies  are  far-reaching,  yet  there  is  little  empirical  data  on 
which  to  base  these  decisions.  Criteria  are  being  developed 
for  determining  eligibility  for  receiving  public  mental  health 
care  and  for  determining  the  levels  of  care  that  will  be  made 
available  to  consumers.  These  summaries  are  intended  to 
provide  examples  of  how  state  administrators  have 
struggled  with,  and  to  some  extent  already  experimented 
with,  various  criteria  and  approaches. 

The  systems  that  are  taking  shape  in  the  following  states 
will  be  described  in  detail:  North  Carolina,  Arizona, 
Louisiana,  and  Massachusetts.  Each  State  is  using  the  Child 
and  Adolescent  Functional  Assessment  Scale  (CAFAS) 
developed  by  Hodges  to  assess  impairment  in  the  youth's 
functioning.  The  CAFAS  provides  a score  reflecting  on  the 
youth's  functioning  in  eight  areas:  school/work,  home, 
community,  behavior  toward  others,  mood,  self-harmful 
behavior,  abnormal  thinking,  and  substance  use.  The 
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caregivers  can  also  be  rated  on  two  scales  that  reflect  on  the 
caregiver's  ability  to  provide  for  the  youth's  material  needs 
and  for  the  youth's  emotional  and  social  needs.  These 
summaries  highlight  the  factors  that  are  being  considered 
for  inclusion  as  eligibility  criteria,  other  than  impairment. 
The  process  by  which  these  guidelines  are  being  developed 
will  also  be  discussed. 

Criteria  for  Accessing  Child  Mental  Health  and 
Substance  Abuse  Services  in  North  Carolina 

Lenore  Behar,  Ph.D.  & Lynn  Stelle 

In  January  1994,  the  North  Carolina  state  office  of  Mental 
Health,  Developmental  Disabilities,  and  Substance  Abuse 
Services  (NIHIDDISAS)  and  the  local  MH/DD/SA 
programs  began  a Medicaid  capitated,  managed  care  waiver 
program  for  children,  under  age  1 8,  with  mental  health  or 
substance  abuse  diagnoses.  Developed  jointly  by  the  state 
and  local  entities,  the  waiver  program,  called  Carolina 
Alternatives,  has  been  piloted  in  32  of  the  state's  100 
counties  with  a plan  to  be  statewide  by  1997.  All  children 
eligible  for  Medicaid  and  in  need  of  treatment  are  to  be 
served.  As  a part  of  the  changes  required  to  operate  the 
waiver,  there  have  been  numerous  revisions  in  policies  and 
procedures  over  the  past  two  years.  During  the  past  six 
months,  a committee  has  drafted  criteria  for  levels  of  care  to 
be  authorized  through  the  managed  care  program. 

As  a second  part  of  the  waiver  program,  in  addition  to  being 
a statewide  program  for  children,  the  capitated  managed 
care  program  will  be  expanded  during  1996-98  to  include 
adults  with  mental  health  and  substance  abuse  problems 
who  are  eligible  for  Medicaid  funded  services.  Thus,  the 
criteria  were  developed  to  apply  to  current  children  and  to 
adults  when  that  population  is  included  in  the  waiver 
program.  The  following  discussion  will  cover  issues  related 
only  to  the  development  and  application  of  the  criteria  for 
levels  of  care  for  children.  The  process  by  which  these 
criteria  are  being  developed  involves  the  following: 

• First,  it  was  important  to  begin  operation  of  the 
program  to  enable  service  providers  to  gain  an 
experiential  understanding  of  the  changes  required  in 
a managed  care  approach  to  service  delivery.  Staff  in 
the  local  MH/DD/SA  programs  gained  familiarity 
with  the  process  of  managing  care,  that  is  determining 
if  services  were  needed  and  what  services  were 
needed  and  then  estimating  the  amount  of  service 
needed  to  address  the  child/adolescent's  mental  health 
or  substance  abuse  problems.  In  addition  to  gaining 
specificity  in  treatment  planning,  the  staff  also  gained 
experience  in  learning  to  authorize,  coordinate,  and 
manage  care  with  networks  of  other  public  and 
private  providers,  including  inpatient  settings.  These 
new  experiences  provided  a foundation  on  which  to 
build  a set  of  criteria  that  could  be  used  to  authorize 
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build  a set  of  criteria  that  could  be  used  to  authorize 
levels  of  care  that  were  related  to  levels  of  intensity  of 
services. 

• A committee  of  clinical  services  providers, 
administrators,  and  members  of  consumer/parent 
advocacy  groups  was  formed  to  develop  the  criteria. 
Starting  with  the  levels  of  care  criteria  developed  in 
the  Fort  Bragg  Project,  the  committee  made  revisions 
they  thought  would  clarify  or  improve  the  Fort  Bragg 
product.  They  added  the  North  Carolina  Functional 
Assessment  Scale  (NCFAS)  for  adults  and  the  Child 
and  Adolescent  Functional  Assessment  Scale 
(CAFAS)  for  children  to  divide  clients  into  six 
categories  of  severity  that  were  to  be  related  to  six 
levels  of  care.  The  product  of  the  committee  was 
reviewed  by  a senior  level  group  composed  of 
directors  of  local  MH/DD/SA  programs. 

• Following  approval  by  this  senior  group,  the  draft 
criteria  were  circulated  to  a wide  range  of 
stakeholders,  including  consumer  groups,  advocacy 
groups,  professional  organizations,  public  and  private 
provider  groups  and  all  local  MH/DD/SA  programs. 
After  receiving  comments  from  this  wide  group  of 
stakeholders,  due  by  the  end  of  February,  a final  draft 
will  be  forwarded  to  the  Director  of  MH/DD/SAS  as 
the  recommended  criteria  for  determining  levels  of 
care.  It  is  planned  that  following  approval  from  the 
Director  and  his  staff,  these  criteria  will  be  used  for 
six  months  for  a period  of  pilot  testing.  After  six 
months  of  use,  revisions  will  be  made,  if  needed,  and 
the  decision  will  be  made  whether  or  not  to  codify  the 
criteria  into  rules. 

A set  of  six  principles  has  been  developed  to  apply  to  all 
levels  of  care.  A summary  of  the  principles  follows. 

• Treatment  must  be  medically  necessary. 

• A deficiency  in  adaptive  functioning  must  be  evident 
and  based  on  clinical  symptoms  related  to  DSM-IV 
diagnoses  approved  for  Carolina  Alternatives. 

• Treatment  is  to  alleviate  problems  associated  with 
DSM-IV,  Axis  I diagnoses  and/or  to  lessen 
manifestations  of  Axis  II  diagnoses. 

• Treatment  is  to  be  provided  in  the  most  clinically 
appropriate  level  of  care  in  the  least  restrictive,  least 
intensive  manner.  Treatment  is  to  be  time  sensitive 
and  strength  oriented  and  should  focus  on  solutions, 
building  on  family  strengths  and  resources. 

• Outcomes  of  treatment  should  be  improved  adaptive 
ability,  prevention  of  relapse,  or  for  emergency 
situations,  stabilization. 

• The  needs  of  the  client  are  the  primary  focus  of 
treatment.  The  convenience  of  the  community,  family 
or  judiciary  are  not  grounds  for  medical  necessity. 

Draft  criteria  have  been  developed  to  address  six  levels  of 
care  ranging  from  preventive  services  to  inpatient  services. 
Each  level  of  care,  except  prevention,  is  accompanied  by  a 
recommended  review  period.  For  each  level  of  care,  criteria 


298 


recommended  review  period.  For  each  level  of  care,  criteria 
are  provided  both  for  admission  to  the  service  and  for 
continuation  of  the  service.  The  levels  of  care  including 
types  of  services  and  recommended  review  periods  follows. 

Prevention:  respite  services,  after-school  services,  drop-in 
services,  screening  or  evaluation;  no  recommended  review 
period. 

Level  A:  group  therapy,  high  risk  intervention-periodic 
(group),  community-based  intervention  (group);  the 
recommended  review  period  is  six  months.  The  latter  two 
categories  are  essentially  wraparound  services  provided  by 
professional  staff  and  by  nonprofessional  staff. 

Level  B:  individual  therapy,  high  risk  intervention 
(individual),  community-based  intervention  (individual);  the 
recommended  review  period  is  three  months. 

Level  C:  partial  hospitalization,  day  treatment,  case 
management;  the  recommended  review  period  is  two 
months. 

Level  D:  moderate  or  intensive  treatment  in  a residential 
setting,  facility-based  crisis  intervention;  the  recommended 
review  period  is  one  month. 

Level  E:  inpatient;  the  recommended  review  period  is  daily. 

The  criteria  for  admission  to  services  at  each  of  the  six 
levels  are  based  on  severity  of  the  child's  problems. 
Continuation  criteria  are  based  on  documentation  of 
continued  need  for  treatment  at  that  level  of  care.  Severity  is 
determined  by  a CAFAS  score  for  all  levels  except;  (a)  the 
minimal  level  which  provides  for  the  delivery  of  prevention 
services  and  (b)  the  maximum  level  which  provides  for  the 
use  of  inpatient  services. 

1.  To  qualify  for  prevention  services,  the  child  or  family 
must  demonstrate  a need  for  such  services  to  prevent  or 
delay  onset  of  psychiatric  problems  in  the  child  or  to 
decrease  the  likelihood  of  onset. 

2.  To  qualify  for  services  at  the  inpatient  level,  in  addition 
to  a CAFAS  score  greater  than  100,  the  child  must:  (1)  have 
a DSM-IV  diagnosis;  and  (2)  must  be  considered  dangerous 
to  him/herself,  dangerous  to  others,  have  a psychosis  or 
other  severely  disorganizing  condition,  or  have  a life 
threatening  medical  condition  (e.g.,  toxic  drug  level),  or 
need  complex  diagnostic  procedures  or  supervision  of 
medication;  and  (3)  constant  and  skilled 
monitoring/treatment  must  be  provided  by  a physician;  and 
(4)  other  less  intensive  services  will  not  suffice. 

As  noted  above,  these  criteria  are  currently  under  review 
and  most  likely  will  be  modified  before  pilot  testing.  Pilot 
testing  is  scheduled  to  last  for  six  months,  or  possibly  more 
if  needed,  so  that  there  is  confidence  in  and  comfort  with  the 
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if  needed,  so  that  there  is  confidence  in  and  comfort  with  the 
application  of  these  criteria  to  clinical  services. 

Proposed  Eligibility  Criteria  and  Procedures  for 
Enrollment  in  Department  of  Mental  Health 
Continuing  Care 

Elizabeth  Irvin,  Ph.D.  & Phyllis  Hersch,  Ph.D. 

Legislation  referred  to  as  Chapter  599  established  the 
primary  mission  for  the  Massachusetts  Department  of 
Mental  Health  (DMH):  "to  provide  services  to  citizens  with 
long-term  or  serious  mental  illness."  This  requires  the 
Department  to  target  programs  and  services  to  persons  with 
long-term  or  serious  mental  illness.  Therefore,  the  primary 
mission  of  DMH  is  to  direct  its  services  to  adults  with 
serious  or  long-term  mental  illness  and  children  and 
adolescents  with  serious  emotional  disturbances.  These 
priority  clients  may  access  continuing  care  services  funded 
by  or  provided  by  the  DMH. 

The  purpose  of  this  summary  is  to  describe  the  clinical 
assessment  criteria  and  procedures  being  proposed  by  the 
Department  of  Mental  Health  to  assess  the  eligibility  of 
children  and  adolescents  for  enrollment  as  priority  clients  in 
DMH  continuing  care  services.  Inclusion  in  the  target 
population  defined  by  these  criteria  is  based  on  the  presence 
of  a serious  emotional  disorder  that  has  resulted  in 
functional  impairment  that  substantially  interferes  with  or 
limits  the  performance  of  one  or  more  major  life  activities. 
These  criteria  integrate  federal  definitions  for  serious 
emotional  disorder  with  the  Department's  current  eligibility 
guidelines  for  children  and  adolescents. 

Youth  applying  for  any  DMH-funded  service  will  be 
assessed  for  their  level  of  need  for  mental  health  services 
using  the  criteria  described  in  this  document.  Based  on  the 
findings  from  the  assessment,  children  and  adolescents  who 
meet  eligibility  criteria  will  be  enrolled  as  a DMH 
continuing  care  client  and  assigned  to  a Care  Management 
site  for  treatment  planning  and  service  authorization.  Care 
management  sites  are  located  in  communities  across  the 
state  and  are  part  of  the  Comprehensive  Community 
Support  System  of  Services.  Care  Management  personnel 
provide  clinical  oversight,  care  planning,  service 
authorization,  service  coordination,  referral,  and  advocacy 
for  eligible,  enrolled  members.  All  DMH  services  are 
authorized  through  Care  Management  and  are  targeted  to 
level  of  need  and  provided  within  available  resources. 

Although  all  youth  whose  continuing  care  services  are 
funded  through  DMH  must  fall  within  the  eligibility  criteria 
described  in  this  summary,  there  is  no  intention  to  restrict 
the  flexibility  or  responsibility  of  the  Department  of  Mental 
Health  or  the  Division  of  Medical  Assistance  to  tailor 
publicly  funded  service  systems  to  meet  local  mental  health 
needs  and  priorities. 
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Proposed  Deflnition  of  Children  and  Adolescents  with  a 
Serious  Emotional  Disorder 

To  be  determined  eligible  for  DMH  services,  children  and 
adolescents  must  have  a diagnosable  mental,  behavioral,  or 
emotional  disorder  that  has  resulted  in  significant  functional 
impairment.  Specifically,  to  qualify  for  continuing  care 
services  through  the  Department  of  Mental  Health,  the 
youth  must  be  1 8 years  of  age  or  younger  at  the  time  of 
application  and  have  a qualifying  mental,  behavioral,  or 
emotional  disorder  of  sufficient  duration  to  meet  criteria 
specified  within  the  Diagnostic  and  Statistical  Manual  of 
Mental  Disorders,  Fourth  Edition  (DSM-IV;  APA,  1994). 
The  qualifying  disorder  must  have  existed  for  at  least  six 
months  prior  to  application  and  be  expected  to  last  for  at 
least  one  year,  and  the  disorder  must  have  resulted  in 
functional  impairment  which  substantially  interferes  with  or 
limits  the  youth's  role  or  functioning  in  family,  school,  or 
community  activities.  The  service  history  of  youth  meeting 
these  eligibility  criteria  will  be  reviewed  prior  to  final 
approval  for  enrollment. 

For  purposes  of  determining  eligibility  for  DMH  services,  a 
child  or  adolescent  with  a serious  emotional  disturbance  is 
one  whose  progressive  personality  development  is  interfered 
with  or  arrested  by  a variety  of  factors  so  that  there  is 
impairment  in  the  capacity  expected,  given  the  child  or 
adolescent's  age  and  endowment.  All  of  these  disorders  have 
episodic,  recurrent,  or  persistent  features;  however,  they 
vary  in  terms  of  severity  and  disabling  effects.  Thus, 
children  and  adolescents  with  serious  emotional 
disturbances  will  have  a diminished  capacity  to  reasonably 
and  accurately  perceive  the  world  around  them,  control  their 
impulses,  maintain  satisfying  or  satisfactory  relations  with 
others,  and/or  to  learn.  Serious  or  long-term  emotional 
disturbances  include  schizophrenia  and  disorders  of  affect 
and  personality,  as  well  as  other  qualifying  disorders 
depending  on  the  severity  and  duration  of  the  illness. 
Children  and  adolescents  with  serious  emotional 
disturbances  may  have  disorders  of  impulse  control  or 
attention  deficit.  However,  all  youth  meeting  the  general 
eligibility  criteria  must  also  meet  the  functional  impairment 
criteria.  The  five  eligibility  criteria  contained  in  this 
definition  are  further  defined  below: 

Criterion  A:  Qualifying  Mental  Disorder.  Qualifying  mental 
disorders  are  all  disorders  listed  in  the  DSM-IV  with  the 
exception  of  those  specifically  excluded.  Excluded  disorders 
are  usually  first  diagnosed  in  infancy,  childhood,  or 
adolescence,  and  include,  but  are  not  limited  to  mental 
retardation,  pervasive  developmental  disorders,  and  autism. 
Also  excluded  are  (a)  conditions  categorized  under 
Delirium,  Dementia,  Amnestic,  and  Other  Cognitive 
Disorders;  (b)  mental  disorders  due  to  a general  medical 
condition;  and  (c)  conditions  listed  under  Substance-Related 
Disorders.  While  mental  health  practitioners  are  frequently 
involved  in  treatment  planning  and  service  delivery  for 
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involved  in  treatment  planning  and  service  delivery  for 
these  children  and  adolescents,  separate  Federal  block  grant 
funds  and  processes  for  needs  assessments  address  these 
population  groups. 

Criterion  B:  Duration  of  Qualifying  Disorder.  The 
qualifying  mental,  behavioral,  or  emotional  disorder  shall 
have  existed  for  six  months  prior  to  application  and  be 
expected  to  last  for  at  least  one  year. 

Criterion  C:  Functional  Impairment.  Functional  impairment 
is  defined  as  difficulties  that  substantially  interfere  with  or 
limit  a child  or  adolescent  from  achieving  or  maintaining 
one  or  more  developmentally  appropriate  social,  behavioral, 
cognitive,  communicative,  or  adaptive  skills.  Functional 
impairment  of  episodic,  recurrent,  and  continuous  duration 
are  included  unless  they  are  temporary  and  expected 
responses  to  stressful  events  in  the  child  or  adolescent's 
environment. 

Operationally,  functional  impairment  related  to  a qualifying 
behavioral,  emotional,  or  mental  disorder  will  be  evaluated 
using  standards  outlined  in  the  Child  Adolescent  Functional 
Assessment  Scale  (CAFAS:  Hodges,  1994).  The  results  of 
an  evaluation  using  these  standards  will  determine  if  the 
applicant  meets  functional  impairment  criteria  for 
enrollment  in  DMH  continuing  care;  specific  impairment 
thresholds  for  eligibility  are  under  study  and  will  be 
described  once  the  Youth  Eligibility  Pilot  is  completed 
(April  15,1996). 

Criterion  D:  Duration  of  Severe  Functional  Impairment. 
This  is  an  indication  of  how  long  the  youth  has  been 
functioning  in  the  severely  impaired  range.  Severely 
impaired  functioning  may  be  a relatively  constant  feature  of 
the  youth's  disorder,  or  may  come  and  go.  The  period  of 
impairment  need  not  be  consecutive.  To  determine  the 
duration  of  severe  functional  impairment,  the  cumulative 
number  of  months  during  the  past  year  that  the  youth  was 
dysfunctional,  as  identified  by  the  CAFAS,  is  utilized. 

Criterion  E:  Service  Utilization.  Applications  will  be 
reviewed  for  the  adequacy  of  services  the  youth  received  in 
the  six  months  prior  to  referral.  Insured  youth  who  meet 
eligibility  criteria  for  Groups  2 or  3,  who  have  not  been 
offered  services  matched  to  their  level  of  need  and  ability  to 
utilize  such  services  during  the  six  months  prior  to  referral, 
may  at  the  discretion  of  the  Department,  not  be  enrolled  in 
DMH-fiinded  services  until  appropriate  interventions  have 
been  tried  in  the  community. 

Procedures 

Standardized  criteria  are  used  to  evaluate  eligibility  for 
DMH  services  and  to  classify  eligible  applicants  into  one  of 
four  clinically  related  groups.  The  criteria  are:  (a)  diagnosis, 
(b)  duration  of  illness,  (c)  severity  of  functional  impairment, 
and  (d)  duration  of  severe  functional  impairment. 
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The  general  approach  is  to  establish  that  the  youth  meets 
clinical  criteria  for  a qualifying  DSM-IV  psychiatric 
diagnosis  and  that  the  youth  is  functionally  impaired  as  a 
result  of  the  qualifying  disorder.  Using  duration  of 
functional  impairment  criteria,  eligible  youth  are  then 
classified  into  one  of  three  clinically  related  groups:  Severe 
and  Persistent  Mental  or  Behavioral  Disorder  (Group  1), 
Severe  Mental  or  Behavioral  Disorder  (Group  2),  and 
Moderately  Severe  Mental  or  Behavioral  Disorder  (Group 
3).  Youth  who  meet  eligibility  criteria  for  Groups  1,  2,  or  3 
are  also  evaluated  with  regard  to  their  prior  services.  Youth 
who  have  not  had  services  matched  to  their  level  of  need 
during  the  six  months  prior  to  referral,  may  at  the  discretion 
of  the  Department,  not  be  enrolled  until  appropriate 
interventions  have  been  tried  in  the  community.  The 
Department  reserves  the  right  to  enroll  persons  who  require 
advocacy  to  access  the  full  benefit  of  their  insurance. 

Eligible  service  recipients  will  be  enrolled  regardless  of 
their  ability  to  pay.  Third  party  reimbursement  or  client 
funds  will  be  accessed  when  service  recipients  have  this 
alternative  available  prior  to  use  of  DMH  funds.  Enrollment 
in  DMH  continuing  care  does  not  constitute  an  entitlement 
for,  or  a guarantee  of  services.  Youth  enrolled  in  DMH  will 
be  served  based  upon  available  resources,  and  services  will 
be  available  to  priority  clients  most  in  need  of  those 
services.  DMH  services  may  be  terminated  once  the  youth 
no  longer  requires  continuing  care  services  using  standard 
criteria. 
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Using  the  Child  and  Adolescent  Functional 
Assessment  Scale  (CAFAS)  to  Establish 
Level-of-Need  for  Medicaid  Managed  Care 
Services 

Randall  Lemoine,  Ph.D.,  Tony  Speier,  Ph.D.,  Sally  Ellzey, 
BCSW,  & Jo  Pine,  MSW 

Over  the  past  two  years,  the  Louisiana  Office  of  Mental 
Health  (OMH)  has  initiated  public  managed  care  practices 
in  an  effort  to  better  manage  costs  and  to  promote  quality  of 
services  for  Medicaid-funded  programs.  These  practices 
have  included  prior-authorization  for  services  based  on 
target  population  eligibility  and  level-of-need  criteria; 
allocation  of  a fixed  amount  of  services  based  on  assessed 
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allocation  of  a fixed  amount  of  services  based  on  assessed 
level-of-need;  and  ongoing  monitoring  of  service  utilization 
and  outcomes.  OMH  initiated  these  practices  statewide  for 
case  management  services  in  1 994  and  recently  designed  a 
significantly  enhanced  program  for  mental  health 
rehabilitation  services,  enabling  a very  flexible  but  managed 
wraparound  package  of  services.  The  CAFAS  was  selected 
as  the  instrument  to  establish  the  initial  level-of-need  and  to 
monitor  changes  in  child/family  need  and  outcomes  over  the 
course  of  service.  Wraparound  service  packages  were 
designed  for  child/families  with  high,  medium,  and  low 
levels  of  need  as  assessed  by  the  CAFAS.  This  summary 
paper  will  discuss  the  selection  and  use  of  the  CAFAS  for 
this  program  which  was  implemented  in  December,  1995. 

Method 

The  Medicaid  Mental  Health  Rehabilitation  (MHR) 
program  was  implemented  statewide  in  Fiscal  Year  (FY) 
1992  as  a fee-for-service  program  serving  both  adults  with 
serious  mental  illness  and  children/youth  with  severe 
emotional/behavior  disorders.  It  was  designed  to 
supplement  the  existing  array  of  OMH  clinical  and  case 
management  services  and  to  offer  a broad  range  of 
treatments  and  supports  in  natural  community  settings  (e.g., 
home,  school,  etc.)  outside  the  clinic  environment.  These 
services  included  both  traditional  clinical  and  more 
contemporary  psycho-  social  rehabilitation  services.  For 
example,  individual,  group,  or  family  psychotherapy  could 
be  offered  as  well  as  psycho-social  skills  training  and  a 
range  of  support  services  (i.e.,  professional  and 
paraprofessional),  as  long  as  these  did  not  duplicate  clinic 
based  services.  OMH  viewed  this  program  as  providing  the 
basis  for  future  development  of  a comprehensive, 
community-based  service  array.  The  program  was  initially 
administered  and  monitored  completely  through  the  state 
Medicaid  agency. 

The  MHR  program  showed  tremendous  growth  over  the 
three  year  period  of  FY  92-95,  increasing  from  an  initial 
$6M  program  to  a $35M  program.  By  July,  1995,  there 
were  about  335  providers  and  nearly  8,000  children/youth 
served  statewide.  Whereas  this  growth  could  have  been 
considered  positive  in  terms  of  increasing  the  amount  of 
mental  health  services  available  to  children/youth  in  the 
state  program,  reviews  indicated  that  there  were  actually 
widespread  program  abuses.  There  was  increasing  concern 
that  this  program  was  on  a "runaway  course."  The  growing 
provider  network  was  largely  inexperienced  with  delivery  of 
individualized,  wraparound  services  in  natural  settings,  and 
many  agencies  appeared  driven  more  by  the  "business 
opportunity"  of  the  program  than  by  an  interest  in  delivering 
quality  care.  Program  reviews  indicated  that  the  service 
population  included  large  numbers  of  children/youth  who 
did  not  meet  target  population  eligibility  criteria  of  serious 
emotional/behavioral  disorder,  and  services  were 
fragmented. 
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During  this  time,  the  state  also  experienced  a severe  budget 
crisis  which  affected  all  Medicaid  programs.  With  regard  to 
the  MHR  program,  the  legislature  slashed  the  program 
budget  for  the  coming  FY  96  from  $35M  to  $12. 9M,  a 63% 
reduction,  and  required  that  it  be  capped  at  that  level 
funding.  The  legislature  further  required  that  the  program 
expenditures  be  audited  monthly  and  gave  the  state 
Medicaid  agency  180  days  to  "fix"  the  program  or  eliminate 
it.  The  fiscal  crisis  and  the  program  abuses  that  preceded 
provided  OMH  and  the  Medicaid  agency  with  an 
opportunity  and  incentive  to  redesign  the  program  utilizing 
public-based  managed  care  practices  to  contain  costs, 
promote  quality,  and  to  protect  the  scarce  dollars  remaining 
for  the  children/youth  most-in-need  of  this  service. 

The  redesign  took  place  very  quickly  in  two  phases,  the  first 
being  the  "quick  fix"  to  gain  immediate  program  control, 
and  the  second,  and  the  subject  of  this  summary,  initiating  a 
more  sound  and  permanent  program  design  that  included 
several  enhancements  to  promote  quality  and  outcome  of 
care.  The  entire  redesign  process  was  done  in  a close 
partnership  with  the  Medicaid  agency,  based  on  the 
foundation  of  the  prior  positive  experience  observed  in  the 
successful  case  management  program.  During  Phase  I of  the 
redesign,  OMH  became  an  administrative  agent  of 
Medicaid,  as  had  been  done  previously  with  case 
management,  and  utilized  its  experienced  regional 
prior-authorization  (PA)  staff  (most  of  whom  were  seasoned 
OMH  clinicians)  to  pre-certify  cases  based  on  explicit  target 
population  eligibility  criteria  (i.e.,  diagnosis,  disability,  and 
duration).  This  step  alone  was  quite  effective  in  assuring 
children/families  most-in-need  were  receiving  services. 

Under  Phase  II,  each  agency  was  to  offer  a comprehensive, 
wraparound  plan  of  professional  and  paraprofessional 
services  through  pre-established  service  packages  based  on 
the  child/family's  assessed  levels  of  need  for  the  service. 
Three  tiers  of  service. packages  were  designed:  High, 
Medium,  and  Low  (i.e..  High  being  intensive  and  Low 
being  the  maintenance  level).  In  order  to  assure  a uniform 
and  complete  data  base  for  target  population  and 
level-of-need  eligibility  determination  by  PA  staff,  OMH 
required  provider  agencies  to  submit  a standardized  clinical 
evaluation  completed  by  both  a psychiatrist/psychologist 
and  social  worker.  Finally,  a case-rate  payment 
methodology  was  developed  to  pay  provider  agencies  for 
the  menu  of  services  delivered. 

A key  component  of  the  redesigned  program  was  the 
assessment  of  level  of  need  (LON).  This  required  an 
instrument  that  would  enable  OMH  regional  PA  staff  to 
reliably  assess  the  child/family  current  service  need  levels 
based  on  data  submitted  by  the  provider.  OMH  needed  an 
instrument  that  was  relatively  brief,  but  comprehensive  and 
sound.  The  instrument  also  needed  to  have  adjustable 
leveling  criteria  and  be  utilized  relatively  objectively,  yet 
still  be  rated  from  clinical  records.  The  development  team 
reviewed  other  states'  experience  in  use  of  LON  technology. 
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reviewed  other  states'  experience  in  use  of  LON  technology. 
Only  a few  states  had  begun  to  employ  LON  technology, 
but  those  that  did  were  utilizing  multidimensional 
level-of-functioning  instruments  to  assess  need  in  addition 
to  target  population  eligibility  criteria.  The  OMH 
development  team  was  impressed  with  North  Carolina's 
Level-of-Eligibility  (LOE)  approach,  using  the  Child  and 
Adolescent  Functional  Assessment  Scale  (CAFAS).  This 
approach  had  an  established  track  record  for  identifying 
eligibility  and  service  levels  along  with  other  pertinent 
factors,  and  the  CAFAS  had  undergone  impressive  recent 
developments  (Hodges,  1994).  In  addition,  the  CAFAS 
included  both  child  and  caregiver  scales,  so  that  family 
issues  could  be  incorporated  into  the  LON  ratings.  After 
some  initial  piloting,  the  CAFAS  (1994  version)  was 
selected  for  use.  In  order  that  the  selection  of  cases  to  be 
served  would  be  sensitive  to  both  child  and  family  needs, 
OMH  decided  to  utilize  the  combined  total  of  the  five  youth 
clinical  and  two  caregiver  scores  as  defining  the  overall 
LON. 

Because  the  program  budget  was  so  limited,  a difficult 
administrative  decision  had  to  be  made  to  first  serve  only 
those  child/youth  that  rated  at  a high  LON.  It  was 
understood  that  if  more  money  was  made  available  and 
added  to  the  program  later,  OMH  would  have  the  ability  to 
serve  persons  at  the  moderate  and  low  levels  of  need  at 
program  entry. 

All  remaining  MHR  cases  (N  = 551)  were  rated  by  PAs 
using  the  CAFAS.  The  distribution  of  scores  was  reviewed 
to  determine  the  cut-off  points  that  would  be  used  for  high, 
medium,  and  low  levels.  In  setting  the  levels  of  need  based 
on  the  CAFAS  total  ratings,  OMH  was  guided  by  what  had 
been  previously  used  by  North  Carolina  for  level  of 
eligibility  criteria  and  by  the  established  CAFAS  guidelines 
for  total  of  the  youth  scales. 

Results 

The  distribution  of  total  CAFAS  scores  (combined  total 
youth  and  caregiver  scores)  was  normally  distributed  and 
showed  a mean,  median,  and  modal  rating  of  90,  with  a 
standard  deviation  of  30.  A total  score  of  90  on  the  youth 
scales  alone  has  been  utilized  in  North  Carolina  for  Level  I, 
the  highest  of  the  four  levels  of  eligibility.  Since  90  was  the 
mean  and  modal  level  for  the  select  group  of  child/youth 
known  to  meet  target  population  eligibility  by 
prior-authorization,  OMH  determined  that  the  high  level 
score  cutoff  would  have  to  be  above  this  level.  A total 
CAFAS  score  of  120  would  be  about  one  standard  deviation 
above  the  mean.  A score  of  60  would  be  one  standard 
deviation  below  the  mean.  So  OMH  decided  on  levels  at 
120  or  above  for  High,  60  to  110  for  Medium,  and  50  or 
below  for  Low.  This  resulted  in  23%  (or  a total  of  120)  of 
then  active  cases  being  classified  as  High  and  eligible  for 
the  program  at  the  time  of  re-implementation.  These  score 
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cutoffs  were  considered  meaningful  by  the  PAs  based  on 
their  personal  experience  with  the  assessment  data  for  cases 
they  were  rating.  The  redesigned  program  was  implemented 
in  December,  1995,  using  these  cutoff  scores. 

Conclusions 

Program  reviews  over  the  early  stages  of  this  program  have 
indicated  that  the  combination  of  target  population 
eligibility  criteria  and  level-of-need,  as  measured  by  the 
CAFAS,  has  been  working  well  in  identifying  those 
most-in-need  for  the  service.  In  addition,  much  is  being 
learned  about  applying  level-of-need  technology.  There 
have  been  ongoing  discussions  regarding  the  concept  of 
LON  and  how  it  may  be  further  developed.  One  important 
consideration  is  that  LON  is  based  solely  on  current  level  of 
functioning  or  impairment,  which  is  subject  to  a variety  of 
influences,  especially  the  existing  level  of  services/supports 
received.  Several  provider  agencies  argued  that 
children/families  with  high-needs  did  not  score  at  the  High 
level  because  they  were  receiving  a level  of  services  at  the 
time  of  the  rating  which  lowered  their  CAFAS  score;  that  is, 
maintained  them  at  the  moderate  or  low  levels.  How  to 
adjust  the  LON  to  accommodate  for  existing 
services/supports  at  the  time  of  program  registration  has 
been  an  issue  under  study.  OMH  has  also  been  studying  the 
use  of  the  Caregiver  Scales  in  the  LON  determination  and 
how  ratings  on  these  scales  interact  with  the  youth  scales. 

Some  agencies  have  argued  that  youth  with  serious 
disturbances  and  fairly  functional  families  do  not  meet  the 
high  level-of-need  criteria.  Some  PAs  have  suggested 
weighting  scales  separately.  Similarly,  some  PAs  have 
recommended  scoring  each  of  the  Role  Performance 
sub-domains  (i.e.,  home,  school,  and  community)  areas 
separately. 

A child  with  a severe  rating  on  two  or  more  of  these 
subscales  is  much  more  in  need  than  one  with  severe  rating 
in  only  one.  Finally,  it  has  been  noted  that  the  CAFAS  does 
not  appear  to  be  as  sensitive  to  high  needs  at  preadolescent 
age  levels  as  it  is  for  adolescents.  This  may  be  attributed  to 
scores  reflecting  impairment  appropriate  for  these  age 
groups.  In  other  words,  there  is  more  opportunity  to  score 
higher  or  be  rated  as  more  impaired  on  the  CAFAS  as  the 
child  grows  older. 

OMH  will  be  continuously  refining  and  improving  this 
program  and  the  level-of-need  methodology  over  the 
coming  year.  The  authors  are  hopeful  that  our  experience 
will  not  only  put  this  program  on  a sound  foundation  but 
also  is  paving  the  way  to  quality,  managed, 
community-based  mental  health  services  for  children/youth 
and  families  in  need  in  Louisiana. 

Criteria  Used  in  Determining  Appropriateness 
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of  Service  Utilization  in  Arizona 

Aimee  Schwartz,  M.D.  & Stephen  Perkins,  M.A. 

For  the  past  several  years,  the  Division  of  Behavioral  Health 
Services  of  the  Arizona  Department  of  Health  Services 
(ADHS/BHS)  has  mandated  the  use  of  the  Child  and 
Adolescent  Functional  Assessment  Scale  (CAFAS;  Hodges, 
1 994)  to  determine  the  need  for  case  management  for 
children  and  adolescents.  A score  of  90  or  higher,  not 
including  the  caregiver  scales,  has  been  used  to  qualify 
children  and  adolescents  for  Intensive  Case  Management 
services.  An  override  is  also  in  place,  in  the  event  the  child 
did  not  qualify  by  a 90  CAFAS  score,  but  requires  case 
management  services.  The  Division  of  Behavioral  Health 
Services  also  uses  the  Arizona  Level  of  Functioning 
Assessment  (ALFA),  which  is  based  on  the  Colorado  Client 
Assessment  Report  (CCAR;  Ellis,  Wilson,  8c  Foster,  1984) 
to  capture  functional  levels  for  both  adults  and  children. 

The  Arizona  behavioral  health  service  system  plans  to 
enhance  the  case  management  process  by  establishing  three 
phases  of  case  management.  The  first  phase  is  provided  for 
children  and  adolescents  in  multiple  child-serving  agencies 
who  have  a history  of  repeated  use  of  restrictive  services 
and  treatment  non-compliance.  The  second  phase  is  for 
those  children  and  adolescents  requiring  ongoing  support  to 
access  treatment  services.  The  third  phase,  case 
coordination,  is  for  those  children  who  are  able  to  improve 
with  brief  intervention,  or  who  will  improve  with  minimal 
intervention. 

Methods/Procedures 

To  better  discriminate  the  child's  level  of  need,  BHS  is 
considering  enhancing  use  of  CAFAS  or  ALFA  scales  with 
the  addition  of  a diagnostic  classification  system  to  suggest 
level  of  functioning,  service  level,  and  treatment 
intervention  for  children  and  adolescents. 

This  usage  would  replicate  the  Adult  Mental  Services  Level 
Checklist,  which  is  an  instrument  that  was  originally 
designed  in  1992  for  determination  of  Adult  Seriously 
Mentally  111  (SMI)  and  service  level  need  for  the  adult 
population.  As  mentioned  above,  the  Arizona  Level  of 
Functional  Assessment  (ALFA)  for  adults  provides  function 
measures  based  on  the  Colorado  Client  Assessment  Report 
(CCAR). 

The  first  step  would  be  determination  of  the  diagnostic 
classification.  The  DSM  IV  Axis  I and  II  principal 
diagnoses  have  been  separated  into  four  categories:  major 
biological  disorders,  disorders  requiring  specialized 
reviews,  other  diagnoses  not  requiring  specialized  reviews, 
and  those  diagnoses  typically  associated  with  children  being 
served  through  other  non-mental  health  related  services. 
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Category  I includes  diagnoses  occurring  with  major 
biological  disorders,  including  Bipolar  Disorders,  Major 
Depression,  Schizophrenia,  and  other  diagnoses  of  a 
biological  nature. 

Categoty  II  includes  those  diagnoses  which  may  require 
specialized  reviews.  Disorders  such  as  conduct,  depressive, 
identity,  and  borderline  personality  may  require  further 
review  for  secondary  diagnostic  codings.  Disorders  falling 
into  this  group  often  coincide  with  other  diagnoses  and 
many  times  require  different  service  interventions  than  the 
Category  I group. 

Category  III  includes  other  diagnoses  that  have  initial  onset 
in  childhood,  and  do  not  require  specialized  reviews  for 
secondary  diagnoses  or  other  symptomatic  reviews. 

The  final  Category,  IV,  includes  those  diagnoses  that  are 
typically  found  in  children  involved  in  other  child-serving 
agencies.  Conduct  Disorder,  although  typically  found  in 
children  served  by  multiple  agencies,  is  classified  only  in 
Category  II  due  to  its  requirement  for  specialized  review. 

The  second  step  of  the  service  level  determination  process 
would  utilize  either  the  ALFA  or  CAFAS  scores.  The 
CAFAS  consists  of  eight  scales  describing  the  youth's 
fimctioning  and  two  that  rate  the  youth's  family/caregiver. 
Six  of  the  CAFAS  scale  (or  five  of  the  ALFA  scores)  would 
be  summed  to  yield  a total  that  is  used  to  help  determine  the 
appropriate  level  of  case  management  services.  When  the 
CAFAS  is  used,  School/Work,  Home,  Community, 

Behavior  Toward  Others,  Substance  Use,  and  Family/Social 
Support  scales  would  be  conceptualized  as  functional 
indicators.  Since  each  scale  is  rated  on  a four  point  scale 
(i.e.,  30,  20,  10,  0),  the  range  of  scores  for  these  six  CAFAS 
scales  is  0 to  180,  with  higher  scores  reflecting  greater 
impairment. 

The  two  remaining  CAFAS  scales  used  to  rate  the  youth 
(i.e.,  Mood/Self-Harm  and  Thinking),  would  not  be 
considered  in  determining  service  level.  Rather,  they  could 
be  used  in  conjunction  with  the  diagnostic  classification  to 
predict  and  aid  in  determining  the  type  of  service 
intervention  rather  than  the  service  level. 

The  three  service  levels  correspond  to  the  three  phases  of 
case  management.  First  level  services,  including  phase  one 
case  management,  are  for  those  children  and  adolescents 
scoring  over  120.  Service  level  number  two,  including  phase 
two  case  management,  are  for  those  children  and 
adolescents  scoring  between  70  and  120.  Those  children 
receiving  under  a 70  functional  score  are  able  to  receive 
level  three  services. 

Assigned  service  levels  are  also  an  option  when  service 
providers  do  not  agree  on  the  predicted  service  level.  This 
allows  for  the  child  to  receive  a different  level  of  case 
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management.  This  overrule  will  be  in  effect  until  such  time 
that  State  officials  determine  the  appropriate  cutoff  scores 
for  the  service  levels. 

Conclusion 

This  service  level  determination  process  will  be  developed 
and  piloted  statewide  for  a six-month  period.  At  the  time  a 
review  of  the  service  utilization  and  grouping  of  children 
based  on  diagnoses  and  functioning  levels  will  be 
conducted.  A review  of  the  "assigned  service  levels"  will  be 
compared  to  the  "predicted  service  levels"  to  determine 
appropriate  cut-off  scores  for  the  three  service  levels.  Six 
month  data  will  be  used  to  determine  appropriateness  of 
CAFAS  scales  versus  ALFA  scales  used  to  determine  the 
three  service  levels.  Cluster  analysis  of  these  scores  will  be 
conducted  based  on  the  "assigned  service  levels."  These 
"assigned  service  levels"  will  then  be  compared  to  the 
"predicted  service  levels"  to  determine  appropriateness  of 
established  cutoff  scores  for  service  levels. 

Determination  of  service  level  is  anticipated  to  provide  state 
reviewers  an  opportunity  to  conduct  desk  audits  of  services 
being  provided  to  groups  of  children  and  adolescents.  In 
addition,  the  process  should  help  case  managers  better 
define  the  array  of  treatment  interventions  that  may  be 
required  for  children  and  adolescents. 
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The  Effects  of  Conflict  and  Marital  Status  on 
Child  Behavior 


Introduction 

Although  divorce  is  generally  considered  a risk  factor  for 
the  development  or  exacerbation  of  children's  behavioral 
and  emotional  problems  (Buchanan,  Maccoby,  & 
Dombusch,  1991;  Fine,  Morelad,  & Sachwebel,  1983), 
family  conflict  and  marital  discord  may  be  equally  or  even 
more  detrimental  to  children's  well-being  (Garber,  1991; 
Long  & Slater,  1988;  Tschann,  Johnston,  Kline,  & 
Wallerstein,  1990;  Wallerstein,  1991).  Thus,  the  effects  of 
marital  conflict  may  supersede  the  benefits  of  a physically 
complete  family  unit.  We  sought  to  replicate  the  findings  of 
these  studies  in  a racially/ethnically  and  socio-economically 
diverse,  urban  child  guidance  setting.  We  hypothesized  that 
family  conflict  would  account  for  significant  variance  in 
predicting  childhood  functioning  over  and  above  the 
influence  of  marital  status  (i.e.,  married,  divorced,  or 
separated)  and  other  demographic  variables. 
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Method 


The  sample  consisted  of  95  children  applying  for  clinical 
services  at  the  Yale  Child  Study  Center  Outpatient  Clinic. 
The  children  ranged  in  age  from  3 to  17  years  (M  = 9.3)  and 
represented  a number  of  racial,  ethnic,  and  social 
backgrounds  tsee  Table  IV  Forty-nine  children  (52%;  24 
boys/25  girls)  were  from  families  in  which  the  custodial 
parents  were  currently  married,  33  (35%;  18  boys/15  girls) 
were  from  divorced  homes,  and  13  (14%;  7 boys/6  girls) 
came  from  homes  in  which  the  parents  were  separated.  In 
total,  three-quarters  (n  = 71)  of  the  children  were  Caucasian 
and  one-quarter  (n  = 24)  minority. 

Following  a phone  intake  interview  during  which  general 
demographic  and  symptom  data  were  gathered,  parents  were 
mailed  and  asked  to  complete  the  Family  Environment 
Scale  (FES;  Moos  & Moos,  1986)  and  the  Child  Behavior 
Checklist  (CBCL;  Achenbach  & Edelbrock,  1983).  During 
an  initial  evaluation  period,  intake  clinicians  assessed  and 
provided  estimates  of  children's  global  functioning  using  the 
Children's  Global  Assessment  Scale  (CGAS;  Shaffer  et  al., 
1983)  and  the  Global  Assessment  of  Functioning  scale 
(GAF;  Diagnostic  and  Statistical  Manual  of  Mental 
Disorders-III-R,  1987). 

Results 

Preliminary  Analyses 

In  order  to  assess  group  (married,  divorced,  separated) 
equivalence  on  demographic  variables,  omnibus  chi-square 
analyses  were  conducted.  Results  indicated  significant 
differences  in  social  class  [Hollingshead  Five  Factor  Index 
of  Social  Status  (1975)  was  collapsed  into  three  categories], 
c2  (4,  N = 91)  = 12.05,  p < .05,  and  geographical  location 
(urban  vs.  suWban),  c2  (2,  N = 95)  = 15.17,  p < .001 . 
Differences  in  ethnic  distribution  (Caucasians  vs. 
minorities)  among  the  three  groups  approached  but  did  not 
reach  statistical  significance,  c2  (2,  N = 95)  = 5.40,  p < .07. 
(see  Table  1 for  specific  cell  counts  and  percentages).  The 
three  groups  did  not  differ  in  terms  of  child  age,  gender,  or 
referral  source  (formal/coercive  vs.  informal/non-coercive; 
see  Table  2). 

In  order  to  compare  the  functioning  of  children  from 
married,  divorced,  and  separated  families,  the  means  of 
these  three  groups  on  the  CBCL,  CGAS,  GAF,  and  Conflict 
subscale  of  the  FES  were  examined  using  Oneway  Analysis 
of  Variance  (ANOVA);  Tukey's  Honestly  Significant 
Difference  Test  was  used  for  comparisons  between  the  three 
groups  (see  Table  2).  Results  indicated  no  statistical 
differences  between  the  three  groups  on  any  measures  as 
rated  by  the  parent  or  clinician.  Furthermore,  there  were  no 
significant  differences  between  minorities  and 
non-minorities  on  these  measures  of  child  behavior, 
functioning,  and  family  conflict. 
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functioning,  and  family  conflict. 

Main  Analyses 

In  order  to  examine  our  primary  hypothesis  that  family 
conflict  would  contribute  unique  variance  in  predicting 
child  functioning  and  symptom  level,  we  conducted 
hierarchical  regression  analyses  in  which  SES,  child  race, 
gender,  and  age  were  analyzed  as  a first  block,  followed  by 
marital  status  in  a second  block,  and  then  family  conflict  on 
block  3.  In  this  way,  we  accounted  for  the  variance 
attributable  to  demographic  variables  and  marital  status 
before  considering  the  influence  of  family  conflict.  Findings 
indicated  that  family  conflict  made  an  additional,  unique 
contribution  in  predicting  child  externalizing  problems  and 
total  symptom  level.  However,  family  conflict  was  not 
predictive  of  internalizing  problems  or  global  functioning  as 
rated  by  the  clinician.  Furthermore,  marital  status  did  not 
make  a significant  contribution  in  predicting  child 
functioning  on  any  of  the  dependent  measures  after 
demographic  variables  were  entered. 

Discussion 

The  current  work  partially  replicates  the  findings  of 
previous  studies  that  demonstrate  the  impact  of  familial 
conflict  on  child  impairment  over  and  above  marital  status. 
Specifically,  we  found  that  family  conflict  predicted  child 
externalizing  behavior  as  reported  by  the  caretaker  (usually 
the  mother).  Family  conflict  was  not  predictive  of 
internalizing  problems  as  reported  by  the  parent  nor  of 
functioning  as  rated  by  the  clinician.  One  might  hypothesize 
that  children  in  stressful  family  situations  often  become  the 
recipients  of  parental  projections  and  thus  become  viewed 
negatively  by  caretakers  (Armbruster,  Dobuler,  Fischer,  & 
Grigsby,  1996).  Although  level  of  conflict  may  have 
influenced  a parent's  perception  of  the  child's  externalizing 
problems,  reported  familial  conflict  did  not  appear  to  affect 
the  clinician's  assessment  of  the  child's  functioning.  This 
finding  once  again  highlights  the  importance  of  considering 
the  informant  when  evaluating  children's  pathology  (see 
Achenbach,  McConaughy,  & Howell,  1 987)  and  appears 
consistent  with  other  studies  which  have  found  that  mothers 
tend  to  over-report  negative  child  behaviors  (Christensen, 
Margolin,  & Sullaway,  1992). 

In  the  current  environment  it  is  important  to  note  that 
minorities  and  non-minorities  in  our  study  did  not  differ  in 
behavior  and  functioning  as  rated  by  caretakers  and 
clinicians.  This  is  important  in  light  of  the  finding  that 
minorities  are  often  perceived  as  having  more  severe 
impairment  than  non-minorities  (Cheung  & Snowden, 

1990).  Future  studies  may  want  to  examine  the  relation  of 
ethnicity  and  child/family  pathology  within  separate  marital 
categories. 

A major  limitation  of  our  study  is  the  absence  of  father  and 
child  reports  of  behavior,  as  well  as  ratings  of  parent 
psychopathology.  Such  information  may  shed  light  on  why 
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psychopathology.  Such  information  may  shed  light  on  why 
caretaker's  perception  of  conflict  predicts  externalizing  but 
not  internalizing  problems.  We  hope  to  include  such 
measures  in  future  studies  that  will  allow  us  to  examine  the 
complex  relationships  between  child  impairment,  different 
informants,  and  caretaker's  psychopathology  across  child's 
gender,  race,  SES,  and  age. 

See  Table  3 for  Regression  Analyses. 
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Introduction 

Managed  mental  health  services,  including  child  mental 
health  services,  are  rapidly  becoming  the  norm  in  the  United 
States.  This  is  particularly  the  case  for  public  child  mental 
health  services  previously  funded  through  Medicaid,  Title 
XIX.  There  are  expectations  of  greater  cost  and  operational 
efficiency  as  well  as  possibly  greater  effectiveness  in  the 
delivery  of  child  mental  health  services  through  this  model. 
On  the  other  side,  there  are  serious  concerns  and 
apprehension  that  this  model  will  be  misapplied,  leading  to 
reductions  in  child  mental  health  services  just  as  they  were 
beginning  to  go  beyond  meager  levels. 

In  this  conversion  to  the  managed  care  model  and  planning 
for  new  service  systems,  little  attention  has  so  far  been  paid 
to  results  from  child  mental  health  epidemiological  and 
services  research.  This  developing  body  of  literature  points 
to  (a)  high  levels  of  mental  health  service  need  in  child  and 
adolescent  populations;  (b)  how  such  need  is  not  addressed 
as  a result  of  a number  of  limitations  in  the  existing  service 
system;  and  (c)  how  children  and  youth  in  need  of  mental 
health  services  are  shifted  into  other  service  sectors  (e.g., 
juvenile  justice,  social  services,  education,  etc.),  defeating 
any  efforts  at  cost  containment.  Whether  the  designers  of 
managed  care  systems  for  children  and  their  families  attend 
to  these  cautionary  tales  will  determine  their  actual  success 
or  failure  in  reaching  their  proposed  outcomes. 

In  this  summary,  findings  from  three  studies  are  briefly 
described  to  illustrate  the  perils  around  which  managed  care 
must  navigate  in  order  to  construct  effective  systems  of  care 
for  children  and  youth. 
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The  Role  of  Diagnostic  Systems 

Theodore  Fallon,  M.D.,  M.P.H.  & Mary  Schwab-Stone, 
M.D. 

This  study  examined  the  perils  involved  in  the  use  of 
categorical  diagnoses  to  identify  who  is  in  need  of  care. 

Methods 

The  study  sample  included  219  school  children  ages  6 to  12 
years  chosen  from  a general  population  in  urban  and  rural 
Eastern  Connecticut.  The  sampling  techniques  were 
designed  to  mimic  a random  sample  of  the  population  over 
sampled  for  psychopathology.  This  epidemiological  survey 
is  remarkable  for  the  acquisition  of  three  informants  for 
each  child  (i.e.,  the  parent,  a teacher,  and  the  child).  Each 
informant  was  interviewed  with  the  Diagnostic  Interview 
Schedule  for  Children;  as  well,  other  instruments  assessing 
the  school,  neighborhood,  and  family  environment  were 
applied.  The  interviewer  also  assigned  a Children's  Global 
Assessment  Scale  (CGAS)  score  to  each  child  interviewed, 
depending  on  their  level  of  function.  The  diagnosis  of 
Attention  Deficit  Hyperactivity  Disorder  (ADHD)  was  used 
to  explore  the  effect  that  informant  disagreement  has  on 
determining  categorical  diagnoses.  The  DSM-III-R  criteria 
of  the  presence  of  at  least  8 of  14  symptoms  was  used  as  the 
definition  of  ADHD.  The  information  from  the  three 
informants  was  then  combined  in  12  different  ways  using 
one,  two,  or  all  three  of  the  informants. 

Results 

Similar  to  the  clinical  situation,  disagreement  among 
informants  was  common  in  this  data  set.  Depending  on 
which  informants  were  queried  and  how  the  information 
from  the  informants  was  combined,  anywhere  from  3%  to 
62%  of  this  general  child  population  sample  could  be 
diagnosed  as  having  ADHD.  This  large  variation  in 
prevalence  occurred  despite  using  the  same  DSM-III-R 
criteria  for  ADHD.  Also,  depending  on  which  algorithm 
was  used,  the  average  CGAS  score  varied  between  55 
(significantly  disturbed)  and  72  (only  mildly  disturbed). 

Discussion 

The  results  of  this  data  analysis  demonstrated  that 
criterion-based  diagnoses  are  limited  in  their  ability  to 
determine  which  children  are  in  need  of  treatment.  In  fact, 
given  the  DSM-III-R  diagnostic  criteria  for  Attention 
Deficit  Hyperactivity  Disorder,  a managed  care  organization 
could  choose  to  provide  services  to  62%  of  a general 
population  or  to  deny  care  to  all  but  3%  of  the  group, 
depending  on  how  they  interpreted  the  information. 
Considering  the  process  at  the  clinician  level,  it  is  important 
for  practitioners  to  recognize  the  diagnostic  ambiguities  that 
remain  even  after  assigning  a diagnosis. 
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The  Role  of  Economics  and  Jails:  Access  and 
Outcome  for  Under-Served  Youth 

Andres  J.  Pumariega,  M.D. 

This  study  illustrates  that  entire  populations  can  be 
overlooked  when  considering  who  needs  care. 

Methods 

This  study  examined  two  population  samples.  The  first 
sample  included  a school-based  sample  of  2585  youth  in  a 
tri-ethnic  population  in  two  regions  of  Texas.  A Youth  Self 
Report  was  used  to  survey  this  group.  The  second  sample 
included  an  incarcerated  youth  population  and  a community 
mental  health  treated  group  of  youths,  both  from  South 
Carolina.  This  second  sample  was  surveyed  using  the 
Diagnostic  Interview  Schedule  for  Children  version  2.3  and 
the  Child  Behavior  Checklist  (CBCL).  Youth  who  met 
diagnostic  criteria  on  one  diagnosis  on  the  Diagnostic 
Interview  Schedule  for  Children  2.3  and  had  a total  t score 
of  greater  than  70  on  the  Youth  Self  Report  were  identified 
as  having  significant  psychopathology.  Both  populations 
were  also  given  questionnaires  eliciting  demographic 
variables  and  mental  health  service  utilization. 

Results 

In  the  first  sample  of  school  youth  from  Texas,  service 
utilization  amongst  this  tri-ethnic  sample  never  rose  above 
30  percent,  even  in  children  with  total  t scores  over  80  on 
the  Youth  Self  Report.  Although  ethnicity  and  access  to 
services  appeared  to  be  the  major  factors  influencing  service 
use,  multiple  regression  analyses  indicate  that 
socioeconomic  factors  were  the  strongest  determining 
variables. 

In  the  second  sample,  the  two  groups  (i.e.,  incarcerated 
youth  versus  community  mental  health  treated  youth)  had 
almost  identical  prevalence  of  significant  psychopathology 
(53%  for  incarcerated  youth  versus  56%  for  community 
treated  youth);  however,  incarcerated  youth  had 
significantly  lower  prior  utilization  of  mental  health  services 
and  higher  utilization  of  long  term  residential  services. 

Discussion 

The  result  suggest  that  youth  from  low  socioeconomic 
populations  are  under-served,  and  that  in  fact  the  poorer 
they  are,  the  less  they  are  provided  for,  even  for  those  youth 
who  had  the  most  severe  disturbances.  The  youth  from  this 
group  with  the  most  severe  disturbances  are  especially 
vulnerable,  and  most  likely  to  end  up  in  the  juvenile  justice 
system.  Results  from  the  second  study  suggest  that  despite 
need  (significant  psychopathology),  these  children  receive 
even  fewer  mental  health  services  and  that  there  is  cost 
shifting  from  mental  health  services  to  juvenile  justice 
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shifting  from  mental  health  services  to  juvenile  justice 
services. 

A Longitudinal  Study  of  Adolescent  Mental 
Health  Service  Use 

Steven  P.  Cuffe,  M.D.,  K.  Jackson,  A.B.,  J.  Waller,  Ph.D., 

S.  Chilappagari,  M.D.,  C.  Z.  Garrison,  Ph.D.,  C.  Addy, 
Ph.D.,  R.  McKeown,  Ph.D. 

There  is  a paucity  of  research  on  the  patterns  of  mental 
health  service  use  in  community  samples  of  adolescents. 
Existing  evidence  suggests  that  most  children  and 
adolescents  with  psychiatric  disorders  receive  no  treatment 
and  that  older  adolescents  are  much  less  likely  to  receive 
treatment  than  younger  children.  We  report  on  mental  health 
service  use  data  from  a longitudinal,  school-based, 
epidemiologic  study  of  adolescent  depression. 

Methods 

From  1986-1988,  a total  of  3283  7th,  8th,  and  9th  graders 
were  screened  with  the  Center  for  Epidemiologic  Studies 
Depression  Scale  (CES-D).  The  top  decile  of  CES-D  scorers 
and  a random  sample  of  the  remainder  were  interviewed 
using  the  Schedule  for  Affective  Disorders  and 
Schizophrenia  for  School-aged  Children  (K-SADS).  Two 
waves  of  interviews  were  performed.  The  first  wave  (N  = 
478)  was  conducted  immediately  following  the  screening 
while  the  subjects  were  ages  12-15  , and  the  second  wave 
(N  = 490)  was  administered  from  1992-1995  while  the 
subjects  were  aged  17-20.  Finally,  a thorough  assessment  of 
service  use  and  perceived  barriers  to  care  was  performed  via 
a telephone  survey  (N  = 344)  and  questionnaire  (N  = 330) 
during  late  1994  and  1995  when  the  subjects  were  aged 
19-22. 

Results 

A dramatic  decrease  in  the  frequency  of  professional  contact 
for  emotional  problems  was  found  over  the  course  of 
adolescence  from  22%  of  the  sample  in  early  adolescence  to 
9.1%  during  late  adolescence  to  3.2%  in  late 
adolescence/early  adulthood.  Caucasians  and  males  were 
more  likely  to  receive  treatment  in  wave  one,  with 
African-American  females  having  significantly  lower 
service  use  in  the  multi-variable  model.  The  differences  in 
service  use  between  races  narrowed  over  the  course  of 
adolescence.  Gender  differences  in  the  use  of  mental  health 
services  also  changed  significantly  over  time.  In  wave  two, 
service  use  was  almost  equal  among  males  and  females, 
while  in  the  most  recent  survey,  females  were  significantly 
more  likely  to  receive  treatment  than  males  (p  = 0.02).  In 
the  barrier  questionnaire,  80%  of  the  respondents  reported 
that  financial  issues  were  a barrier  to  obtaining  treatment. 
Over  half  the  sample  stated  a preference  for  using  family, 
friends,  ministers,  or  family  doctors  for  help  with  their 


emotional  problems.  Finally,  approximately  one  third  of  the 
sample  endorsed  privacy  or  stigma  issues  and  lack  of 
knowledge  of  where  to  obtain  help.  In  a regression  analysis, 
it  was  demonstrated  that  African-American  females  report 
significantly  more  money  barriers,  females  report 
significantly  fewer  work  barriers,  and  Caucasians 
significantly  more  knowledge  barriers. 

Discussion 

These  data  suggest  that  most  adolescents  with  psychiatric 
disorders  fail  to  receive  treatment.  There  are  significant 
differences  in  both  service  use  and  perceived  barriers  to 
services  by  race  and  gender.  There  is  a clear  need  for 
increased  accessibility  and  cultural  sensitivity  in  the 
provision  of  mental  health  services.  There  are  many  lessons 
and  dilemmas  posed  for  managed  care  companies  by  these 
data.  Under-use  of  services  by  adolescents  experiencing 
emotional  or  behavioral  problems  may  reduce  the 
immediate  costs  to  health  care  companies.  However,  the 
long-term  functioning  of  these  individuals  may  be  poor, 
resulting  in  far  greater  costs  to  the  companies  over  time, 
assuming  a no  eject,  no  reject  policy  requiring  companies  to 
take  responsibility  for  the  health  care  of  their  customers. 
Outcome  data  is  needed  to  evaluate  the  efficacy  of 
treatment,  and  to  assess  productivity,  as  well  as  overall 
service  use  costs.  Research  in  the  area  of  early  intervention 
and  case  detection  is  sorely  needed.  Managed  care 
companies  should  also  investigate  the  significant  decrease 
in  service  use  in  late  adolescence  and  take  steps  to  reduce 
the  barriers  to  care  for  this  group.  Finally,  consumers  and 
advocacy  groups  should  demand  improved  accessibility  and 
culturally  sensitive  care. 

Conclusions 

The  developing  systems  of  managed  mental  health  care  hold 
important  potential  for  systems  of  care  for  children  and 
adolescents.  With  coordination  of  efforts  and  intelligent 
allocation  of  services,  children's  service  systems  hold  great 
promise.  However,  these  three  studies  demonstrate  that 
there  are  many  perils  facing  the  development  of  managed 
mental  health  care.  If  we  close  our  eyes  to  these  perils, 
efforts  to  provide  efficient  and  effective  care  for  children 
have  a significant  potential  to  go  awry. 
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What  the  Literature  Tells  Us:  Relationships 
Between  Family  Violence,  School  Behavior 
Problems,  Juvenile  Delinquency,  and  Adult 

Crime 


Introduction 

Childhood  victimization  and  the  witnessing  of  violence  in 
the  family  have  substantiated  short-  and  long-term  negative 
consequences  for  child  and  adult  behaviors.  The  deleterious 
physical  and  emotional  impact  on  children  and  families  of 
violence  in  the  home,  community,  and  school  is  a serious 
problem.  Correlations  between  family  violence,  child  abuse 
and  neglect,  school  behavior  problems,  juvenile 
delinquency,  and  criminal  activity  have  been  documented. 
However,  there  is  limited  empirical  knowledge  in  each  of 
the  areas  and  even  less  when  inter-relationships  among  the 
areas  are  explored.  The  purpose  of  this  paper  is  to  construct 
a multi-systemic  picture  of  the  impact  of  family  violence  on 
children,  families,  and  society  through  the  empirical 
literature  in  these  areas  and  their  inter-relationships. 

A review  of  the  literature  is  important  as  a framework  upon 
which  to  construct  future  research,  practice,  and  policy.  A 
search  of  the  psychological,  sociological,  educational, 
medical,  and  dissertation  indexes  of  literature  over  the  past 
15  years  provided  a broad  representation  of  the  empirical 
research  on  family  violence.  Additional  research  is  needed, 
especially  prospective  studies  that  consider  questions 
regarding  methodological  issues. 
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Violence  in  the  Family 

The  "cycle  of  violence,"  "violence  begets  violence,"  and 
"intergenerational  transmission  of  violence"  are  phrases  that 
are  often  used  interchangeably  in  the  literature  to  refer  to 
children  who  were  abused  becoming  abusers  or  violent 
offenders,  and  more  recently,  witnesses  of  family  violence 
becoming  victims  or  violent  offenders.  Although  most 
survivors  of  childhood  maltreatment  do  not  have  a criminal 
record,  a longitudinal  study  of  abuse  demonstrated  that 
those  who  were  abused  or  neglected  as  children  were  58% 
more  likely  to  be  arrested  as  juveniles,  38%  more  likely  to 
be  arrested  as  adults,  and  38%  more  likely  to  be  arrested  for 
a violent  crime  (Widom,  1989). 

The  term  family  violence  has  often  been  used  to  focus 
specifically  on  marital  violence.  Researchers  and 
practitioners  have  broadened  the  field's  scope  and  the 
public's  awareness  by  identifying  family  violence  as  all 
violence  that  occurs  in  the  family.  Family  violence  includes 
child  abuse  and  neglect,  spouse  abuse,  sibling  abuse, 
courtship  abuse,  elder  abuse,  and  violence  witnessed  in  the 
home.  The  following  sections  describe  the  relationship  of 
many  of  the  manifestations  of  family  violence  to  a variety  of 
behavioral,  emotional,  and  functional  problems. 

Child  Abuse  and  Neglect 

The  experience  of  physical  abuse  has  been  associated  with 
aggression  toward  others.  Children  who  have  been  sexually 
abused  have  been  shown  to  exhibit  more  maladaptive  sexual 
behavior,  and  children  who  have  been  psychologically 
abused  experience  low  self-esteem  (Briere  & Runtz,  1988; 
Claussen  & Crittenden,  1991;  Copeland,  1988).  Infants  and 
young  children  who  have  been  physically  abused  have  been 
found  to  play  less  than  comparison  nonabused  infants,  be 
less  focused,  show  less  positive  affect  (Howard,  1986; 
Wasserman,  Gardier,  Allen,  & Shilansky,  1987),  and  show 
patterns  of  anxious  or  disorganized  attachment  (Cicchetti, 
1987;  Field,  1987).  Sexual  incest  has  been  linked  to 
depression,  nightmares,  bed  wetting,  clinging,  and  anxiety 
(Oates,  1987),  and  neglect  has  been  connected  with 
emotional  withdrawal  and  intellectual  delay  (Kent,  1976). 

Marital/Parental  Violence  and  Child  Maltreatment 

Rosenbaum  and  O'Leary  (1981)  found  that  82%  of  the  men 
in  their  study  who  had  witnessed  parental  spouse  abuse 
were  also  victims  of  child  abuse.  Lewis  (1987)  indicated 
that  women  who  were  physically  abused  by  their  partner  in 
adulthood  were  five  times  more  likely  than  a comparison 
group  of  nonabused  women  to  report  experiences  of  child 
abuse. 

It  has  been  observed  that  children  of  battered  women  are  at 
an  increased  risk  for  being  abused,  with  estimates  of  an 
overlap  between  spousal  abuse  and  child  abuse  ranging 
from  30%  to  60%  (Hughes,  1988;  O'Keefe,  1995;  Suh  & 


from  30%  to  60%  (Hughes,  1988;  O'Keefe,  1995;  Suh  & 
Abel,  1990).  Children  who  were  witnesses  to  marital 
violence  in  addition  to  being  victims  of  abuse  have  been 
discovered  to  have  the  greatest  behavior  problems  when 
compared  to  children  who  only  witnessed  violence  or  those 
with  neither  experience  (Hughes,  1988;  Kalmuss,  1984). 
Some  of  their  behavior  problems  include  conduct  problems 
(Hershom  & Rosenbaum,  1985),  adjustment  difficulties 
(Jaffe,  Wolfe,  Wilson,  & Zak,  1986),  and  aggression 
(O'Keefe,  1995).  Increases  in  sleep  disorders,  abdominal 
distress,  headaches  and  backaches,  and  obesity  were  found 
to  be  physical  consequences  of  family  violence  (Browne  & 
Finkelhor,  1986;  Moeller,  Bachmann,  & Moeller,  1993; 
Rimza,  & Niggermann,  1 982).  Children  exposed  to  family 
violence  have  also  been  found  to  exhibit  fewer  interests  and 
social  activities  and  lower  school  performance  (Wolfe,  Zak, 
Wilson,  & Jaffe,  1986). 

Family  Violence  and  School  Behavior  Problems 

Violence  in  schools  has  become  an  increasing  problem,  and 
solutions  have  not  been  readily  visible.  Problems  have 
included  but  are  not  limited  to  truancy,  assault  and  battery 
(on  other  children  and  staff),  substance  use,  bullying,  sexual 
offenses,  homicide  and  suicide,  gang  violence,  and  racially 
motivated  violence.  Many  familial  and  environmental 
factors  have  been  studied  to  explain  or  predict  these  school 
behavior  problems. 

Children  who  have  been  physically  abused  have  been  found 
to  display  poor  school  adjustment  and  academic 
performance  (Garbarino,  1987;  Kurtz,  Gaudin,  Wodarski,  & 
Howing,  1993).  Children  who  have  been  physically 
neglected  have  been  reported  to  manifest  more  behavior 
problems  and  have  greater  school  and  academic  difficulties 
than  control  groups  (Garbarino,  1987;  Reidy,  1977). 
Children  who  have  been  subjected  to  maltreatment 
demonstrate  a higher  risk  of  failure  in  school,  and  older 
children  (age  14  and  up)  who  are  experiencing  external 
problems  (e.g.,  pregnancy,  institutionalization,  etc.)  often 
end  up  dropping  out  (Kurtz  et  al.,  1993).  Trupin,  Tarico, 
Low,  Jemelka,  and  McClellan  (1993)  suggested  that  the 
cognitive  and  emotional  sequelae  of  abuse  contribute  to 
severe  emotional  disturbances. 

Zingraff,  Leiter,  Johnsen,  and  Myers  (1994)  found  that 
adequate  school  performance  was  associated  with  a 
substantial  reduction  in  the  risk  of  delinquency  among 
maltreated  children,  particularly  those  who  suffered  from 
parental  neglect.  This  is  one  of  the  few  studies  that  has 
looked  at  the  relationship  between  child  maltreatment, 
juvenile  delinquency,  and  school  behavior. 

Violence  in  the  Family  and  Juvenile  Delinquency 

Patterson  (1982)  hypothesized  that  the  reciprocal  interactive 
processes  of  child,  parent,  and  external  stress  were 
significant  in  producing  aggression,  child  abuse,  and 
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significant  in  producing  aggression,  child  abuse,  and 
delinquency.  Empirical  evidence  from  the  self-reports  of 
adolescents  who  are  delinquents  concerning  prior  abuse 
histories  have  resulted  in  rates  from  21%  (Alfaro,  1981)  to 
51%  (Mouzakitis,  1981).  Adolescents  who  had  committed 
sexual  offenses  were  often  found  to  have  experienced 
physical  and  sexual  abuse  as  well  as  neglect  (Widom  & 
Ames,  1994).  Tarter,  Hegedus,  Winsten,  and  Alterman 
(1984)  found  that  44%  of  the  children  in  their  study  who 
had  been  abused  committed  violent  crimes  of  an  assaultive 
nature,  compared  with  16%  of  the  nonabused  children  who 
were  delinquents. 

Scudder,  Blount,  Heide,  and  Silverman  (1993)  reported  that 
students  with  delinquency  referrals  had  higher  rates  of  abuse 
(21.4%)  than  students  who  had  no  delinquency  referrals 
(6.1%).  Empirical  evidence  has  also  associated  childhood 
abuse  with  teenage  runaways,  with  percentages  ranging  as 
high  as  78%  (Farber,  Kinast,  McCoard,  & Falkner,  1984). 
Dembo,  Williams,  Wothke,  Schmeidler,  and  Brown  (1992) 
documented  that  family  problems  and  abuse/victimization 
experiences  influence  youths'  initial  involvement  in  drug 
use  and  delinquent  behavior,  and  that  once  these  behavior 
patterns  were  established,  they  tended  to  continue  over  time. 

Family  Violence  and  Adult  Criminal  Behavior 

Dutton  and  Hart  (1992)  studied  incarcerated  men  and  found 
that  69%  of  those  who  had  witnessed  inter-parental  abuse 
committed  physical  crimes  and  25%  of  those  who  had 
witnessed  inter-parental  abuse  committed  sexual  crimes. 
Among  these  incarcerated  men,  physical  abuse  in  childhood 
was  found  to  increase  the  chances  for  physical  abuse  in  the 
family  by  a factor  of  five,  and  these  men  were  twice  as 
likely  to  abuse  strangers  or  non-family  members.  Similarly, 
being  a victim  of  child  sexual  abuse  increased  the 
probability  of  committing  sexual  abuse  against  strangers 
five  times  and  within  the  family  eight  times  (Dutton  & Hart, 
1992).  Widom  (1989)  discovered  that  the  adults  who  had 
been  abused  and  neglected  as  children  had  a higher  arrest 
rate  for  adult  criminality  and  violent  criminal  behavior. 
Straus  (1991)  found  a correlation  between  laws  authorizing 
physical  punishment  in  schools  and  the  homicide  rate. 

Implications 

Our  society  struggles  to  deal  with  problems  of  negative 
behaviors,  crime,  and  substance  abuse  knowing  that 
experiences  in  the  early  years  often  portend  the  occurrence 
of  these  behaviors  in  later  life.  Children  who  have 
experienced  violence  in  the  family  have  difficulty  relating  to 
others,  school  experiences,  and  standards  and  laws.  These 
children  exhibit  inappropriate  internalizing  and 
externalizing  behaviors  that  impede  their  physical  and 
emotional  development.  The  increase  of  violence  in  our 
communities  has  been  directly  associated  with  intra-familial 
factors.  Research  has  shown  the  adverse  impact  of  child 
maltreatment  and  domestic  violence  not  only  on  the  victims. 
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maltreatment  and  domestic  violence  not  only  on  the  victims, 
but  on  society  as  a whole.  The  cost  to  the  public  is  great, 
financially  and  psychologically.  Intervention  and  prevention 
must  be  linked  to  these  findings. 

Although  the  research  strongly  indicates  that  violence  in  the 
family,  particularly  child  abuse  and  neglect,  is  predictive  of 
future  violent  behavior,  the  majority  of  abused  and/or 
neglected  children  do  not  become  violent  or  victims.  Are 
there  protective  factors  that  exist  in  a child's  life  that  enable 
that  child  to  be  resilient?  Research  is  needed  to  evaluate  the 
potential  mediating  variables  that  act  to  shield  children 
exposed  to  violence  from  later  dysfunctional  behaviors. 

While  there  has  been  an  abundance  of  individual  studies  on 
violence  in  the  family,  and  its  negative  effects  on  children 
have  been  well  established,  findings  have  been  limited  by 
the  diversity  of  subject  characteristics,  definitions  of 
variables,  and  measurement  methods  used. 

Integration  of  these  studies  has  also  been  limited.  The 
difficulty  in  empirically  observing  families  unless  they  are 
willing  participants  (or  a part  of  the  official  records  of  a 
public  agency),  seems  related  to  issues  of  privacy,  trust  or 
potential  for  criminal  prosecution. 

It  should  be  recognized  that  it  is  difficult  to  empirically 
observe  families,  to  match  samples  in  a controlled  study,  to 
find  the  funding  for  longitudinal  research,  and  to  overcome 
the  issues  of  confidentiality  that  impact  the  availability  of 
records.  Long-term  consequences  of  childhood 
maltreatment  have  been  difficult  to  pinpoint,  particularly 
considering  the  many  other  mediating  factors  which  may 
affect  children's  lives.  However,  the  family  is  primary  in  the 
nurturing  of  mentally  healthy  individuals  and  providing  a 
basis  of  knowledge  for  the  causes,  dynamics,  and 
ameliorating  factors  of  family  violence  that  may  impact  the 
resolution  of  many  of  the  problems  faced  in  our  society. 
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1 

A Historical  Perspective  on  Multisystem  Youth 
and  Patterns  of  Institutional  Care 


Introduction 

Much  attention  has  been  given  to  the  complex  nature  of 
childhood  and  the  efforts  of  multiple  service  systems  in 
serving  troubled  youth.  Of  interest  to  this  study  were  those 
youth  with  emotional  disorders  who  were  in  the  care  of  the 
juvenile  justice  system  in  Maryland. 


A historical  analysis,  as  a component  of  a research  plan,  was 
the  initial  step  to  systematically  examine  the  interface  of  the 
Maryland  Juvenile  Justice  Department  and  Mental  Hygiene 
Administration  in  their  efforts  to  provide  services  to  youth 
involved  with  the  juvenile  justice  system  who  were  found  to 
have  mental  disorders.  This  analysis  contributes  to  the 
development  of  a framework  for  common  understanding 
from  which  these  agencies  can  address  future  issues. 
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Method 


Data  Sources 

Utilization  data  were  available  through  the  Maryland 
Mental  Hygiene  Administration  and  document  youth 
receiving  institutional  services  by  lead  agency.  Youth  with  a 
lead  agency  of  either  Mental  Hygiene  or  Juvenile  Justice 
were  included  (N  = 1429).  Computerized  data  of  this  type 
were  limited  (i.e.,  available  from  1993  through  1995). 
Utilization  data  obtained  from  reports  were  available  from 
1987. 

Primary  archival  data  (in  the  form  of  photos,  hospital 
ledgers,  memorandums,  and  legislative  reports)  were 
obtained  through  the  Maryland  State  Archives,  the  Johns 
Hopkins  Medical  Library  Archives,  the  Maryland  Mental 
Hygiene  Administration,  and  the  Maryland  Department  of 
Juvenile  Justice.  Secondary  data  sources  included  reference 
books,  periodicals,  and  legislation  listed  on-line  (i.e., 
Internet).  Evaluation  of  the  historical  data  judged  the 
authenticity  and  content  of  available  materials. 

Analysis 

A time  line  was  constructed  documenting  developments 
from  the  1 6th  century  to  present  day.  Notation  included 
major  social  events  (e.g.,  wars  and  economic  depressions), 
major  legislation,  and  evolution  of  treatment  philosophies 
over  time.  More  recent  history  was  highlighted  by  archival 
photos  taken  from  the  Maryland  Lunacy  Commission 
created  in  1886  by  the  Maryland  State  Legislature.  This 
Commission  was  directed  to  "visit  and  inspect  all  places 
public  or  private,  where  insane  persons  [and  incorrigibles] 
were  kept." 

Patterns  of  institutional  care  were  selected  as  indicators  of 
the  interface  between  the  two  systems  because  of  the 
availability  of  the  data  and  the  use  of  institutional  care 
throughout  the  long  history  of  social  efforts  to  control 
"incorrigibles." 

Results 

Several  themes  repeatedly  become  evident  in  the  historical 
legacies  of  juvenile  justice  and  mental  health  systems.  The 
balance  of  the  societal  need  for  safety  with  the  treatment 
needs  of  the  youth  are  marked  by  advocacy  and  legal  efforts 
and  also  influenced  by  the  economies  and  public  opinion  of 
the  time.  As  an  example,  the  philosophical  conflict  between 
institutional  approaches  and  more  "homelike"  placements 
was  reflected  in  the  "orphan  train"  movement  versus 
development  of  "Houses  of  Refuge"  in  the  1870's  and  again 
a century  later  as  efforts  were  made  to  shift  care  from 
institutional  to  community-based  services. 

Utilization  patterns  for  both  public  and  private  facilities 
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Utilization  patterns  for  both  public  and  private  facilities 


revealed  that  between  1987  and  1994  inpatient  episodes 
decreased  about  40%  in  the  mental  health  sector,  while 
increasing  approximately  the  same  amount  in  the  juvenile 
iustice  sector  ("see  Fieure  ll.  Little  change  in  admissions  to 
private  facilities  has  been  seen  over  time,  with  slightly 
fewer  admissions  by  juvenile  justice  (-4%)  and  no  change  in 
the  mental  health  sector.  For  public  facilities,  however, 
admissions  dropped  markedly  for  mental  health  (-46%),  but 
increased  for  the  juvenile  justice  sector  (+26%).  Admissions 
by  age  demonstrate  a marked  increase  in  adolescent 
admissions  for  both  public  and  private  agencies  (94%  were 
1 3 years  or  older),  with  a greater  number  of  African 
American  youth  admitted  to  Juvenile  Justice  (46%)  than  to 
Mental  Health  (32%).  The  average  length  of  stay  (115  days) 
decreased  for  both  the  public  (-4%)  and  private  (-12%) 
facilities  in  both  sectors  with  the  greatest  reduction 
occurring  in  the  private  sector. 

Conclusion 

The  struggle  to  provide  services  to  these  multisystem  youth 
has  been  a product  of  political-legal  debate  and  economic 
compromise  revolving  around  basic  issues  of  humanitarian 
reform,  security,  and  social  order.  From  the  development  of 
the  first  juvenile  court  in  1899,  the  use  of  the  court  system 
to  access  services  has  contributed  to  the  trends  found.  One 
can  only  conclude  that  the  cyclical  pattern  of  treatment 
versus  punishment  will  continue  as  it  has  since  history  has 
been  recorded. 

We  can  optimize  the  quality  of  services  if  we  were  to 
recognize  the  importance  of  each  child  and  family  unit  to 
the  foture  of  our  society.  Thinking  in  these  terms  would  call 
upon  us  to  join  in  partnerships  with  children  and  families, 
with  our  co-workers  and  our  peers  creating  a system  of  care 
that  balances  the  needs  of  the  individual  with  the  needs  of 
society.  As  professionals  we  need  to  listen  to  children  and 
families,  and  as  advocates,  we  need  to  articulate  our  visions 
for  this  system  of  care. 
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System  of  Care  for  Youth  with  Severe 
Emotional  or  Behavioral  Disorders:  A 
Developmental  Model 


Introduction 

The  current  system  of  care  literature  identifies  core  values 
and  guiding  principles  for  developing  an  ideal  system  of 
care,  often  within  a framework  of  a continuum  of  care,  in 
terms  of  the  degree  of  restrictiveness  of  the  settings  in 
which  children  and  youth  with  severe  emotional  disturbance 
are  placed  (Stroul  & Friedman,  1988).  Friedman  and  Kutash 
(1992)  have  argued  that  one  of  the  challenges  in  the  child 
mental  health  field  is  to  shift  to  an  early  intervention  focus. 
Identifying  and  intervening  in  children's  problems  at  earlier 
ages  may  make  it  possible  to  modestly  loosen  the  grip  that 
poverty  and  social  class  have  on  their  future  prospects 
(Schorr,  1991).  Kahn  and  Kammerman  (1992)  further 
argued  that,  in  theory,  one  of  the  options  for  designing  a 
more  powerful  system  of  care  is  to  emphasize  the  role  of 
socialization  and  child  developmental  programs,  versus 
treatment  services. 

This  summary  introduces  a framework  for  conceptualizing  a 
system  of  care  built  on  a child  development  perspective, 
^^at  is  the  rationale  for  conceptualizing  and  designing  a 
system  of  care  in  this  developmentally  sensitive  manner? 
First,  such  an  approach  focuses  on  developmental 
transitions  which  all  children  and  youth  experience.  Such 
transitions  include  entry  into  school,  negotiating  the 
challenges  associated  with  the  teen  years  in  modern  society, 
and  emancipation  from  home.  In  addition,  these  transitions, 
normal  for  many  children  and  youth,  are  overlaid  by 
additional  transitional  events  such  as  divorce,  remarriage, 
and  the  death  of  a family  member.  When  added  to  routine 
transitions,  these  events  pose  additional  adaptive  burdens 
which  place  such  children  at  risk  for  emotional  and/or 
behavioral  disturbance.  Furthermore,  the  literature  clearly 
shows  that  poverty  is  highly  correlated  with  mental  health 
problems  for  both  children  and  adults.  This  cluster  of 
factors,  which  this  model  seeks  to  highlight,  provides 
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insights  into  the  dynamics  by  which  children  become  at  risk 
for  severe  emotional  or  behavioral  disturbance. 

Throughout  the  developmental  stages,  a system  of  care  built 
around  a developmental  model  would  help  identify  key 
actors  and  systems  of  social  service  delivery.  Beyond 
identifying  key  actors,  however,  it  is  also  necessary  to 
describe  those  key  systems  which  impact  most  heavily  on 
both  healthy  and  clinically  identified  children  and  youth. 

For  example,  NACTS  found  that  the  participants  in  its 
clinical  sample  typically  used  multiple  systems  of  care, 
which  included  schools,  mental  health,  child  welfare, 
corrections,  and  vocational  rehabilitation.  However, 
children  at  different  periods  in  the  developmental 
continuum  tended  to  demonstrate  different  patterns  of 
service  system  usage.  For  example,  in  the  NACTS  study, 
most  children  initially  received  service  from  the  mental 
health  system  and  schools.  Later,  many  of  these  children 
came  in  contact  with  the  child  welfare  system  and  the 
correctional  system.  Indeed,  the  transition  that  occurs  when 
children  move  from  status  as  clients  in  the  mental  health 
system  to  being  relabeled  criminally  deviant  by  the 
correctional  system  warrants  further  analysis  and  policy 
development.  This  is  particularly  true  for  those  children 
who  are  at  high  risk  for  coming  into  contact  with  the 
criminal  justice  system,  such  as  clinical  groups  diagnosed 
with  conduct  disorder  and  attention  deficiencies  disorder. 
Job  placement  programs  (e.g.,  vocational  rehabilitation, 
etc.)  represent  a potential  resource  in  facilitating  the  last 
transition  in  a child's  developmental  period,  resulting  in 
emancipation  from  one's  family. 

England  and  Cole  (1992)  note  that  service  delivery  should 
be  organized  to  respond  consistently  to  the  ever  changing 
needs  of  young  clients  who  are  growing  and  developing. 
Duchnowski  and  Friedman  (1990)  further  argue  that  there  is 
a need  for  viewing  the  community  system  as  the  unit  of 
analysis  in  research,  rather  than  simply  looking  at  small 
components  of  the  system.  A service  system  organized 
around  developmental  principles  addresses  both  of  these 
concerns  by  emphasizing  a community  wide  continuum  of 
care  conceptualized  across  the  developmental  period,  from 
birth  to  twenty-two  of  age. 

The  first  tier  of  a model  for  a developmentally  focused 
system  of  care  includes  a continuum  of  services  which 
focuses  on  those  services  and  policies  which  promote  the 
health  development  of  our  nation's  children  and  youth.  This 
continuum  would  include  both  health  and  social  service 
providers  (e.g.,  parental  care,  well  baby  clinics,  day  care, 
head  start,  schools,  churches,  family  planning  clinics,  drug 
prevention  programs,  and  vocational  rehabilitation).  These 
programs  primarily  focus  on  non-clinically  impaired 
children  and  youth,  but  often  include  children  who  are  at 
serious  risk  for  emotional  and  behavioral  disorders.  As  a 
result,  caretakers  such  as  parents,  clergy,  school  personnel, 
day  care  facilities,  and  primary  care  physicians,  become  key 
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actors  in  the  lives  of  these  children.  This  first  level  in  this 
model  performs  the  important  function  of  promoting  good 
health  and  mental  health,  while  attempting  to  prevent 
serious  mental  health  problems  before  they  arise.  Early 
identification  and  appropriate  referral  become  crucial  tasks 
for  these  actors  and  systems. 

The  second  tier  in  this  model  describes  the  continuum  of 
services  for  those  children  who  have  received  clinical 
diagnoses.  This  continuum  would  include  such  programs  or 
systems  as  The  Early  Prevention  Screening  and  Diagnostic 
Treatment  (EPSDT)  program,  special  education  classes,  a 
range  of  mental  health  programs  from  outpatient  to 
residential  treatment  programs,  and  programs  for  specific 
sub-populations,  such  as  adolescents  with  chemical 
dependency  and  eating  disorders. 

Key  Transitions  and  Implications 

Early  Childhood 

This  model,  by  virtue  of  focusing  on  the  entire 
developmental  period,  focuses  attention  on  early  stages  of 
development,  where  child  mental  health  problems  might  be 
identified  and  addressed,  while  they  may  be  still 
manageable.  NACTS  data  indicate,  for  example,  that  child 
mental  health  problems  were  identified  for  nearly  half 
(48.3%)  of  their  sample  children  prior  to  entry  into  school 
(Newcomb  et  al.,  1995),  highlighting  the  importance  of 
early  identification  and  intervention.  Furthermore,  parents 
were  the  people  most  likely  to  first  notice  that  a problem 
exists  in  their  sample  (57.2%).  Pediatricians  and  family 
practitioners  are  also  potential  sources  of  early 
identification,  but  they  have  been  under-utilized.  Less  than 
two  percent  of  the  children  in  the  NACTS  sample  had 
problems  first  noticed  by  doctors.  Similarly,  clergy  are  an 
under-utilized  resource  for  early  identification  of  problems. 
These  actors  should  be  involved  in  this  process  at  the 
community  level  through  in-service  trainings  provided  by 
mental  health  practitioners.  Psychiatrists,  for  example,  may 
be  more  viable  trainers  for  medical  personnel,  with  other 
mental  health  professions  providing  such  workshops  to 
other  identified  actors  in  the  community. 

School  Entry 

The  next  major  developmental  transition  centers  on  entry 
into  school,  and  the  subsequent  expansion  of  the  child's 
social  world,  providing  children  both  opportunities  and 
developmental  challenges.  Knoff  and  Batsche  (1990)  argue 
that  schools  must  be  viewed,  in  a developmental  sense,  as  a 
primary  intervention  environment  for  all  children,  especially 
for  those  who  exhibit  educational  and  mental  health 
problems  or  concerns.  One  such  effort  at  strengthening 
services  to  children  and  youth,  within  these  almost  universal 
and  natural  settings,  is  a major  initiative  of  the  Florida 
Department  of  Education  to  develop  "full  service  schools" 
in  which  many  mental  health  services  are  located  on  site 
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in  which  many  mental  health  services  are  located  on  site 
(Florida  Mental  Health  Institute,  1991).  Morse  (1993)  has 
provided  a model  of  the  roles  schools  could  play  in  the 
prevention  of  mental  health  problems  in  children  and  youth. 

The  Teen  Years 

These  children  and  youth  then  are  required  to  negotiate 
transition  from  elementary  schools  to  middle,  junior,  and 
senior  high  schools.  At  the  present  time  in  the  United  States, 
there  is  an  alarmingly  high  level  of  interrelated  negative 
outcomes  for  young  people.  For  many  teens,  these  outcomes 
include  dropping  out  of  school,  substance  abuse,  aggressive 
behavior,  emotional  disturbance,  teen  pregnancy, 
unemployment,  and  criminal  behavior  (Florida  Mental 
Health  Institute,  1991).  The  increased  complexity  of  the 
adolescent's  social  world,  in  combination  with  the  multiple 
crises  that  often  accompany  this  developmental  time  frame, 
place  tremendous  burdens  on  the  multiple  systems  in  which 
these  children  are  engaged,  as  well  as  on  the  families  and 
children,  themselves.  Thus,  it  is  imperative  that 
communities  and  families  need  to  marshal  all  of  the 
resources  available  to  them  as  well  as  to  develop  new 
programs  and  strategies.  A critical  systems  issue  during  this 
developmental  period  centers  on  the  transition  of  many 
youth  from  receiving  treatment  in  the  mental  health  system 
to  being  processed  through  the  juvenile  corrections  system. 
Over  one  million  children  and  adolescents  a year  are 
processed  through  the  juvenile  courts,  many  of  whom  are 
children  with  serious  emotional  and/or  behavioral 
disturbance. 

Entering  Adulthood 

The  last  major  transition  these  youth  experience  is  the 
transition  to  adulthood.  Youth  with  clinical  diagnoses  often 
have  a particularly  difficult  task  in  moving  towards 
independence.  Unfortunately,  this  is  an  area  in  which  these 
youths  are  typically  underserved,  and  they  under-utilize 
those  services  which  are  available.  Indeed,  Stroul  and 
Friedman  (1988)  identify  support  for  the  transition  to 
adulthood  as  one  of  their  ten  principles  underlying  an 
optimal  system  of  care.  They  recognize  that  children  who 
"age  out"  of  the  system  of  care  become  young  adults  who 
often  need  long-term  mental  health  care,  vocational 
services,  and  a range  of  other  support  services.  Thus,  it 
becomes  important  to  develop  functional  linkages  with 
relevant  adult  agencies. 

Conclusion 

A developmentally  focused  system  of  care  would  have  the 
advantage  of  building  on  current  trends  in  the  recent 
paradigm  shift  in  child  and  adolescent  mental  health  and 
would  support  principles  of  care  already  described  in  the 
literature.  Costello  et  al.  (1993)  note,  for  example,  that  "in 
the  past  decade,  child  psychiatric  epidemiology  has  started 
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to  look  at  both  public  health  and  causal  aspects  of 
epidemiology  from  a developmental  point  of  view." 

Stroul  and  Friedman  (1988)  articulated  both  core  values  and 
guiding  principles  for  an  ideal  system  of  care.  This  child 
developmental  model,  with  its  emphasis  on  the  role  of 
parents  and  its  community  focus,  is  consistent  with  both  of 
their  core  values.  Specifically,  their  emphasis  on  early 
identification  and  intervention  and  on  a smooth  transition 
into  the  adult  service  system  as  they  reach  maturity  are 
issues  that  a developmentally  focused  system  of  care  would 
at  least  attempt  to  address.  Similarly,  the  National 
Commission  on  Child  Welfare  and  Family  Preservation 
argued  that  states  should  offer  a three  part  system  of  care  for 
children  and  families.  The  first  component  would  involve  a 
universal  infrastructure  for  all  families  and  include  such 
services  as  prenatal  care,  parenting  education,  and 
developmental  services.  A strong  pro  active  family  policy 
would  be  a necessary  and  potentially  powerful  base  upon 
which  to  build  a functional  child  mental  health  system 
attuned  to  a developmental  focus.  This  summary  has 
provided  a beginning  framework  for  such  a system  of  care. 
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Introduction 
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Cleo  Wallace  Center  (CWC),  like  many  other  residential 
psychiatric  facilities  is  finding  third-party  payers  (i.e., 
insurance  companies,  departments  of  social  services)  and 
referral  sources  increasingly  interested  in  outcome  tools  that 
measure  the  effectiveness  of  treatment.  Third  party  payers 
are  also  seeking  the  least  expensive  care  option  (often 
translated  into  least  restrictive  care  setting)  for  children  and 
adolescents.  In  this  climate,  service  providers  must  find 
ways  to -attract  and  retain  , customers  and  demonstrate  that 
the  services  being  provided  are  appropriate,  effective,  and 
cost-efficient. 


In  order  to  address  these  questions,  the  Child  and 
Adolescent  Functional  Assessment  Scale  (CAFAS)  was 
identified  as  the  tool  that  would  work  best  with  our  facility 
(Hodges,  Hickman,  & Kurtz  1991).  The  CAFAS  was  chosen  | 
I because  of  it's  multidimensional  nature  (five  separate 
I subscales  and  a total  score),  the  ease  of  scoring,  inexpensive 
j start  up  costs,  and  understandable  training  materials.  In 
j addition,  the  scores  are  based  on  a child's  observable, 
i specific  behaviors. 


Method 


Starting  in  1994,  all  out-of-state  patients  admitted  into  the 
residential  program  after  January  1 were  scored  at 
admission  and  at  discharge.  Scores  were  assigned  by 
residential  clinicians.  After  eight  months  (September,  1 994), 
all  residential  clients  were  scored  at  admission  and  at 
discharge.  In  January,  1995,  all  day  treatment  clients  were 
added  to  the  list  of  patients  to  be  scored.  A tracking  system 
was  set  up  to  notify  residential  clinicians  when  a CAFAS 
needed  to  be  scored  for  a client  (either  at  admission  or  at 
discharge).  Beginning  in  January,  1 996,  all  patients 
admitted  to  CWC  will  be  assigned  a CAFAS  score  at 
admission  and  at  discharge,  providing  a larger  sample  of  the 
800  patients  a year  admitted  with  lengths-of-stay  from  3 to 
25  days. 

Subjects 

Admission  scores  were  collected  on  213  patients.  A total  of 
102  of  those  patients  also  had  discharge  scores. 

The  average  age  for  all  patients  with  admission  scores  was 
13.8  years,  with  a range  from  3.4  years  to  20.8  years. 
Eighty-eight  girls  and  125  boys  had  admission  scores. 
Sixty-four  percent  of  the  patients  with  scores  were 
Caucasian;  17%  were  African-American;  12%  were 
Hispanic;  and  8%  were  of  some  other  ethnic  background. 
Over  half  (56%)  of  these  patients  were  Colorado  residents. 
The  other  patients  came  from  Nevada  (13%),  Pennsylvania 
(13%),  Nebraska  (10%),  or  other  states  (8%). 

Patients  with  both  CAFAS  scores  had  an  average  age  of  14 
years.  Of  these,  54%  were  boys,  and  46%  were  girls.  A total 
of  69%  were  Caucasian;  13%  were  African  American;  1 1% 
were  Hispanic;  and  7%  were  some  other  ethnic  background. 
The  average  length-of-stay  was  5 months,  with  a range  from 
4 days  to  17  months. 

Twenty-eight  patients  with  both  CAFAS  scores  who  had 
lengths-of-stay  under  3 1 days  were  considered  separately. 

An  updated  analysis  will  be  done  on  patients  with  short 
lengths-of-stay  (3  to  25  days)  after  January  1,  1996,  when 
implementation  of  the  CAFAS  in  the  hospital  program 
begins. 

Measure 

The  CAFAS  is  a multidimensional  tool  used  to  record  the 
extent  to  which  a youth's  mental  health  disorder  is 
disruptive  of  functioning  in  each  of  five  psycho-social  areas; 
(a)  role  performance,  (b)  thinking,  (c)  behavior  towards 
others,  (d)  moods,  and  (e)  substance  use.  For  each  item,  a 
score  is  assigned.  A total  score  is  calculated  by  adding  all 
subscale  item's  scores. 

Two  versions  of  the  CAFAS  tool  were  used  during  this 
study.  The  first  version  did  not  include  subscale  items  for 
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study.  The  first  version  did  not  include  subscale  items  for 
Role  Performance  (school,  home,  community)  or  Moods 
(emotions  and  self  harm).  Moods  and  Role  Performance 
scores  were  calculated  for  the  48  revised  versions  by  taking 
the  highest  of  the  subscale  items. 

Procedures 

Residential  clinicians  were  assigned  to  every  residential  or 
day  treatment  patient  at  the  time  of  admission,  and  were 
determined  to  be  most  familiar  with  each  patient  and  thus 
best  able  to  accurately  assign  individual  CAFAS  scores. 
They  were  responsible  for  therapy  (i.e.,  individual,  group, 
and  family),  treatment  program  planning,  and  working  with 
families  and  caseworkers  on  discharge  plans.  Since  January, 
1994,  20  residential  clinicians  (5  men  and  15  women,  all 
with  Master's  level  backgrounds)  have  assigned  CAFAS 
scores.  Ten  of  these  residential  clinicians  were  required  to 
attend  a two-hour  training  session  on  using  the  CAFAS.  The 
other  ten  were  hired  later  and  received  supervision  in  the 
use  of  the  CAFAS. 

Each  residential  clinician  was  required  to  assign  a CAFAS 
score  to  patients  after  admission  to  the  residential  or  day 
treatment  program.  The  expectation  was  that  the  scores 
would  be  assigned  within  three  weeks.  Forty-six  percent  of 
the  scores  were  assigned  within  three  weeks.  Some  youth 
had  been  in  the  hospital  program  prior  to  transfer  into  the 
residential  or  day  treatment  program,  and  such  patients  were 
not  assigned  a CAFAS  score  at  admission  to  the  facility  but 
rather  when  they  were  transferred  into  the  residential  or  day 
treatment  program.  It  is  not  currently  known  how  long  after 
admission  to  the  residential  program  these  patients  were 
assigned  a score. 

At  the  time  of  discharge  from  the  facility  (or  within  one 
week),  residential  clinicians  were  expected  to  assign 
discharge  scores.  Only  twenty-three  percent  of  discharged 
patients  were  assigned  a discharge  score  within  one  week. 
Sixteen  percent  were  assigned  scores  within  two  weeks. 

Analysis 

Frequencies  and  comparison  of  means  tests  were  used  to 
analyze  the  315  scores  collected  between  January,  1994  and 
September,  1995.  Mean  scores  were  compared  by  gender 
and  race  with  no  statistically  significant  differences  found. 
Additional  means  testing  will  be  done  by  age, 
length-of-stay,  state  of  residence,  placement  before  and  after 
treatment,  and  diagnosis  for  the  final  presentation  of  this 
study.  Paired  sample  t-tests  were  used  to  test  for 
significance  in  paired  scores  (admission  and  discharge). 

Results 

Initial  results  using  the  CAFAS  were  encouraging;  patients 
typically  had  a total  admission  CAFAS  score  of  over  80 
indicating  a need  for  intensive  services.  The  total  average 
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indicating  a need  for  intensive  services.  The  total  average 
score  was  90.1.  Figure  1 describes  the  average  scale  scores 
at  admission  for  213  patients. 

Patients  admitted  to  CWC  were  most  likely  to  show 
disruptions  in  function  for  Role  Performance  (74% 
demonstrated  severe  levels  of  impairment),  Behavior 
Towards  Others  (35%  showed  severe  levels  of  impairment), 
or  Moods  (36%  demonstrated  severe  levels  of  impairment). 
Figure  2 illustrates  the  percentage  of  patients  with  a severe 
impairment  rating  at  admission  and  discharge. 

Statistically  significant  differences  were  found  for  all 
subscale  scores  and  for  the  total  CAFAS  score  for  102 
patients  with  both  scores  (p  < .01).  Figure  3 describes  the 
admission  and  discharge  scores. 

Discussion 

Patients  admitted  to  CWC  residential  programs  have  an 
average  TOTAL  score  on  the  CAFAS  of  90.1  indicating  a 
need  for  intensive  services  (i.e.,  the  kind  of  services 
available  in  CWC's  residential  treatment  program).  The 
highest  average  scores  were  in  Role  Performance,  Behavior 
Towards  Others,  and  Moods.  Patients  admitted  to  CWC's 
residential  program  most  often  have  mood  disorders  (e.g., 
depression,  dysthymia,  etc.)  or  behavioral  disorders  (e.g., 
ADHD,  oppositional  defiant  disorder,  impulse  control 
disorder,  etc.),  which  suggests  face  validity  in  describing  the 
population  being  served. 

Average  scores  show  a statistically  significant  change  at 
discharge,  indicating  that  patients  are  improving  their  levels 
of  functioning  during  treatment.  The  group  of  28  patients 
with  lengths-of-stay  under  31  days  also  showed  statistically 
significant  changes  in  their  scores  before  and  after  treatment 
in  the  residential  program  with  the  exception  of  Thinking, 
suggesting  that  the  CAFAS  can  detect  changes  in  levels  of 
functioning  independent  of  their  length-of-stay. 

We  expect  to  find  a relationship  between  length-of-stay  in 
the  inpatient  program  and  the  degree  to  which  change  is 
made  in  the  levels  of  functioning  on  the  CAFAS.  This  may 
allow  us  to  describe  to  third  party  payer  sources  what  kinds 
of  lengths-of-stay  and  costs  they  can  expect  for  patients 
with  different  levels  of  impairment.  For  instance,  we  expect 
to  find  that  suicidal  patients  needing  crises  stabilization  over 
a short  length-of-stay  will  show  marked  improvement  on  the 
Moods/Self  Harm  scale  but  not  necessarily  any  change  in 
Behavior  Towards  Others.  Using  data  from  the  CAFAS,  we 
may  be  able  to  tell  payers  what  treatment  benefit  (based  on 
functional  areas)  they  can  expect  to  see  with  different 
lengths-of-stay.  This  will  allow  payer  sources  to  more 
accurately  plan  for  lengths-of-stay  and  benefits  accordingly. 
Recidivism  rates  may  also  be  affected  by  more  effective  and 
accurate  planning  based  on  CAFAS  scores  and 
lengths-of-stay. 
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Two  concerns  emerged  through  the  analysis  of  this  data. 
First,  raters  need  to  be  more  quickly  prompted  after 
admission  and  after  discharge  to  assign  scores.  This  will  be 
addressed  with  the  introduction  of  a computerized  software 
program  that  allows  clinicians  to  enter  scores  directly  into  a 
database  with  a reminder  feature.  This  protocol  is  scheduled 
for  implementation  when  the  hospital  will  be  added  to  the 
pool  of  patients  being  assigned  CAFAS  scores.  Second, 
while  inter-rater  reliability  appears  to  be  acceptable,  not  all 
staff  have  been  formally  trained  and  inter-rater  reliability 
tested.  This  will  be  addressed  when  all  staff  required  to 
assign  CAFAS  scores  will  be  trained  in  an  eight-hour 
training  program. 
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Introduction 


The  impetus  toward  providing  community  based  care  is  a 
powerful  driving  force  in  the  managed  care  revolution 
impacting  changes  in  the  child  welfare  and  mental  health 
systems.  Unfortunately,  there  are  still  children  who  require 
intensive  treatment  environments  in  order  for  the  treatment 
to  be  effective  in  assessing,  stabilizing,  and  treating  highly 
disruptive  and  potentially  dangerous  behaviors.  Some 
researchers  suggest  that  intensive  treatment  environments 
such  as  inpatient  or  residential  are  not  necessary  if 
home-based  intervention  is  available  (Stroul  & Friedman, 
1986;  Tuma,  1989  Weithom,  1988),  and  that  these 
environments  may  be  detrimental  to  the  future  functioning 
of  youth  (Henggeler,  1989).  Others  propose  that  inpatient 
treatments,  particularly  those  with  specialized  treatment 
interventions,  may  be  a beneficial  option  (Pfeiffer  & 
Strzelecki,  1990).  While  intensive  treatment  such  as 
residential  care  may  be  a viable  treatment  option  for  some 
high-risk  youth  (United  States  General  Accounting  Office, 

1 994),  not  enough  is  known  about  these  programs  to 
provide  a clear  picture  of  which  kinds  of  treatment 
approaches  work  best  with  different  subpopulations  of 
youth  with  severe  emotional  disabilities  (SED).  Current 
understanding  is  limited  and  primarily  anecdotal  regarding 
how  best  to  combine  intensive  treatment  environments  with 
home-based  care  alternatives  to  serve  the  most  at-risk  youth 
over  time.  Programs  with  a diverse  continuum  of 
service-intensive  care  that  serve  youth  through  a variety  of 
different  models  and  environments  may  be  in  a strong 
position  to  shed  light  on  this  question. 

It  is  evident  that  intensive  treatment  environments  can  no 
longer  be  conceptualized  as  stand  alone  services;  rather, 
they  must  be  conceptualized  and  operated  as  part  of  a highly 
integrated  continuum  of  care  to  serve  high-risk  youth  and 
their  families  over  time.  This  continuum  of  care  may 
provide  the  necessary  ingredients  to  maximize  treatment 
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effect,  particularly  through  the  interaction  between 
wraparound  services  and  specialized  programmatic  milieus. 
Anecdotal  clinical  experience  and  data  suggest  that  there  are 
still  important  practice  issues  and  dynamics  that  need  to  be 
studied  and  resolved  before  intensive  treatment  milieus  can 
be  completely  removed  from  the  child  mental  health 
continuum. 

Method 

Specific  behavioral  characteristics  of  children  in  four 
different  treatment  environments  were  examined  in  order  to 
shed  light  on  placement  decisions  in  relation  to  levels  of 
care  and  severity  of  symptoms.  Data  were  collected  from 
approximately  300  total  program  admissions  above  age  5, 
each  quarter,  for  an  18  month  period  of  time.  Each  child's 
scores  on  the  Devereux  Scales  of  Mental  Disorders  (DSMD: 
Naglieri,  LeBuffe,  & Preiffer,  1994)  were  averaged  by  scale, 
composite,  and  total  score  reflecting  their  behavior 
throughout  the  specified  level  of  placement.  The  children 
were  in  four  different  treatment  environments  at  Beech 
Brook,  a child  mental  health  agency.  The  four  programs 
included:  an  intensive  pre-adolescent  age  residential 
treatment  program;  a pre-adolescent  age  day  treatment 
program;  an  urban  (Cleveland  Public  Schools) 
pre-adolescent  age  school  based  mental  health  program  for 
children  with  severe  behavioral  handicaps  (SBH);  and  a 
treatment  foster  care  program.  The  sample  included  90 
children  in  residential  treatment,  51  children  in  a day 
treatment/partial  hospitalization  program,  24  children  in  an 
urban  school-based  mental  health  treatment  program,  and  a 
total  of  166  pre-adolescent  and  adolescent  youngsters  in  a 
treatment  foster  care  program.  Of  this  sample  of  33 1 
children,  61%  were  male,  39%  female,  and  74%  were 
African-American  (see  Table  1).  The  primary  focus  of  this 
investigation  was  to  determine  if  the  type  and  severity  of 
child  behavioral  characteristics  as  measured  by  a 
standardized  behavioral  rating  instrument  accurately 
discriminated  children  in  different  intensive  treatment 
environments.  A secondary  focus  of  this  investigation  was 
to  explore  whether  these  findings  could  help  identify 
clinical  and  practice  issues  inherent  in  treating  the  most 
"deep-end"  youth,  many  of  whom  have  had  access  to  an 
array  of  community  wraparound  service  options. 

The  DSMD  is  a 1 10-item  (or  1 1 1 -item  for  the  adolescent 
version)  behavior  rating  scale  designed  to  evaluate 
behaviors  related  to  psychopathology  in  children  and 
adolescents.  There  are  two  separate  instruments,  with 
comparable  items  and  subscales&endash;one  for  children 
ages  5-12  and  the  other  for  children  ages  13-18.  The 
instrument  has  three  composite  scores  (externalizing, 
internalizing,  and  critical  pathology)  and  each  is  made  up  of 
two  subscales  (conduct  and  attention  [or  delinquency  for 
adolescents],  depression  and  Emxiety,  acute  problems  and 
autism).  An  overall  total  test  T-score  is  also  included.  The 
DSMD  total  and  composite  scales  have  excellent  internal 
reliability  (Cronbach's  alpha  of  .97  for  Total  Scale  score) 
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reliability  (Cronbach's  alpha  of  .97  for  Total  Scale  score) 
and  test-retest  reliability.  Also,  there  is  good  interrater 
reliability  with  a clinical  population  (Cronbach's  alpha  of 
.52  for  Total  Scale  score).  The  DSMD  is  well  suited  for  use 
in  evaluating  mental  health  treatment  outcome  reflecting  the 
full  range  of  psychopathology,  including  the  more  severely 
disturbed  behaviors  that  are  often  missing  from  other  rating 
scales.  A total  test  T-score  of  60  has  been  empirically 
determined  to  be  the  best  cut-score  for  differentiating 
clinical  from  non-clinical  samples. 

Results 

Preliminary  analyses  focused  on  examining  comparisons 
among  youth  between  four  different  levels  of  care. 
Univariate  Analyses  of  Variance  (ANOVAs)  tests  were 
performed  for  the  six  scale,  three  composite,  and  total  scale 
DSMD  scores.  If  the  ANOVA  was  found  to  be  significant 
(p  < .05),  post-hoc  Bonferroni  tests  were  performed  to 
identify  where  the  differences  between  levels  of  care 
resided. 

The  DSMD  mean  T-scores  for  children  in  residential  care 
were  consistently  greater  than  for  subjects  in  other  levels  of 
care  tsee  Figure  1)  and  reliably  discriminated  children  in 
residential  care  versus  treatment  foster  care  on  four 
subscales  (acute,  anxiety,  depression,  and  conduct),  two 
composite  scales  (internalizing  and  critical  pathology),  and 
the  total  scale  T-score  tsee  Table  2).  The  sequencing  of  the 
discrimination  of  children  followed  the  expected  direction 
with  children  in  the  most  intensive  treatment  environments 
having  higher  scores  (indicative  of  greater  dysfunction)  and 
the  children  in  the  less  intensive  treatment  environments 
having  lower  scores.  The  DSMD  further  identified  specific 
problem  domains  (e.g.,  depression)  related  to  the  most 
high-risk  youth  placed  in  residential  treatment.  On  the 
average,  these  youngsters  (both  male  and  female)  tended  to 
be  significantly  more  depressed  than  their  counterparts  in 
day  treatment  and  foster  care.  Depression,  however,  is  not 
the  reason  that  pre-adolescent  children  are  referred  to  the 
most  intensive  treatment  environments.  It  is  typically  for 
more  acute  behaviors  and  the  failure  of  intensive 
community-based  models  to  safely  manage  these  behaviors. 

Perhaps  the  most  concerning  findings  were  that  the 
youngest  children  in  residential  care  (i.e.,  9 years  of  age  or 
less)  were,  overall,  the  most  clinically  disturbed  across 
DSMD  measures,  and  as  a group  had  experienced  the 
highest  rate  of  previous  placement  moves.  Independent 
t-tests  yielded  statistically  significant  differences  on  all 
DSMD  scales  (except  conduct  and  anxiety)  between  the 
younger  residential  children  (age  9 or  less)  versus  the  older 
residential  children  (ages  10  - 13;  see  Table  3).  This 
subgroup  (i.e.,  ages  5-9)  contains  some  of  the  "youngest 
and  the  deepest"  children  in  Ohio's  child  welfare  and  mental 
health  system  for  whom  previous  community  treatment  has 
been  a series  of  disrupted  placement  failures.  As  a whole, 
this  group  appears  to  be  poorly  understood,  highly  under 
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this  group  appears  to  be  poorly  understood,  highly  under 
serviced,  and  systematically  targeted  for  less  intensive 
treatment  options. 

Discussion  and  Implications 

The  implications  of  this  exploratory  study  are  both  practical 
and  theoretical.  Practical  implications  from  this  study 
suggest  that  children  in  different  levels  of  intensive  care 
(i.e.,  residential  and  treatment  foster  care)  can  be 
distinguished  by  the  severity  of  their  behavioral 
characteristics.  It  would  also  appear  that  some  of  the 
"youngest  and  the  deepest"  of  these  children  may  be  at  the 
highest  risk  for  having  treatment  needs  neglected  or  delayed 
until  they  have  experienced  numerous  treatment  failures  or 
dangerous  levels  of  symptom  severity.  While  the  purpose  of 
this  investigation  was  to  identify  characteristics  of  the  role 
that  intensive  treatment  environments  may  play  in 
continuums  of  community-based  care,  controlled  studies  are 
needed  to  determine  the  relative  efficacy  of  these  programs 
compared  to  alternative  treatment  models.  Caution  needs  to 
be  exercised,  however,  in  randomly  assigning  children  with 
the  most  severe  disturbances  due  to  four  issues:  safety 
concerns,  the  high  number  of  previous  treatment  failures, 
the  consequences  of  continued  placement  failures,  and  the 
impact  of  the  subjects'  deteriorating  clinical  course. 

Theoretical  implications  point  to  the  utility  of  blending 
wraparound  and  intensive  programmatic  models  of  service 
to  reduce  disruptions,  stabilize  and  support  placements,  and 
effectively  treat  youth  with  chronic,  severe  impairments. 

The  differential  management  and  treatment  of  these  children 
requires  a broad  range  of  specialized  intervention  options 
addressing  varying  levels  of  impairment  and  need  that  are 
not  quickly  or  easily  remedied.  The  tendency  to  separate 
wraparound  models  of  intervention  from  programmatic 
models  with  highly  individualized  and  specialized  services 
dichotomizes  complex  treatment  realities  and  distorts  the 
development  of  best  clinical  practice  protocols.  The  next 
generation  of  treatment  paradigms  for  youth  with  severe 
emotional  disabilities  will  blend  the  flexibility  of 
wraparound  services  with  the  episodic  intensity  of 
specialized  clinical  paths  and  programs. 
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Introduction 


The  recidivism  of  delinquent  youth  and  characteristics  of 
these  repeat  offenders  have  been  the  subject  of  several 
investigations.  Among  the  characteristics  that  have  been 
found  to  be  correlated  with  repeat  offending  have  been 
attachment  to  parents,  parental  supervision,  alcohol  and 
drug  use,  and  past  criminality  (Loeber  & 
Stouthamer-Loeber,  1986),  in  addition  to  the  demographic 
characteristics.  However,  the  VcU'iables  in  most  of  these 
studies  have  generally  been  limited  to  a few  demographic 
variables  such  as  age,  ethnicity,  and  gender. 

Although  there  is  considerable  information  in  the  literature 
on  the  chau-acteristics  of  youth  detained  in  correctional 
facilities  who  are  repeat  offenders,  there  is  little  information 
in  the  literature  youth  with  mental  illness  who  are  repeat 
offenders.  It  is  unclear  from  the  literature  whether  the 
i detainment  of  youth  with  mental  illness  is  related  to 
^ demographic,  clinical,  or  service  use  characteristics. 
Multiple  investigators  have  found  that  minority  youth  tend 
to  be  incarcerated  at  a higher  rate  than  white  youth 
(Eisenman  & Kritsonis,  1993;  Brown,  Rhodes,  Miller,  & 
Jenkins,  1 990).  This  trend  appears  to  remain  constant  for 
youth  with  mentally  illness  (Lewis,  Shanok,  Cohen, 
Kligfeld,  & Frisone,  1980;  Cohen  et  al,  1990). 

Despite  the  fact  that  there  have  been  few  studies  which 
examine  the  effect  of  gender  on  detention,  gender  bias  also 
appears  to  exist.  Males  tend  to  be  incarcerated  at  a higher 
rate  and  commit  more  serious  offenses  than  females. 
Females,  however,  cU'e  more  likely  to  be  arrested  for  status 
offenses  than  males  (Dembo,  Williams,  & Schmeidler, 
1993).  Female  detainees  also  have  an  elevated  degree  of 
deprivation  and  neglect  at  home  (Chesney-Lind  & Shelden, 
1992). 

j It  has  been  estimated  that  as  many  as  60%  of  youths 
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detained  in  juvenile  corrections  facilities  suffer  from 
emotional  disorders  (Cocozza,  1991).  Youth  detained  in 
corrections  facilities  have  been  found  to  have  levels  of 
psychopathology  similar  to  the  levels  of  mental  illness 
found  in  psychiatric  hospitals  (Davis,  Bean,  Schumacher,  & 
Stringer,  1991;  Pumariega,  in  press).  Despite  these  findings, 
most  U.S.  juvenile  detention  facilities  do  not  have  an 
adequate  screening  procedure  for  detecting  emotional 
disturbance  among  its  detainees  (Barton,  1976).  The 
emotional  difficulties  experienced  by  these  youth  are  likely 
to  receive  little  professional  attention  by  mental  health 
practitioners  prior  to  incarceration.  This  is  a trend  that  is 
likely  to  continue  once  the  youth  is  detained  due  to  the 
paucity  of  mental  health  services  that  are  provided  in  most 
juvenile  justice  settings  (Anno,  1984). 

Furthermore,  the  decision  of  whether  to  provide  treatment 
or  not  is  usually  based  on  the  conclusion  of  an  individual 
staff  member  who  may  not  have  a theoretical  basis  for 
judgement  (Barton,  1976).  Many  of  these  individuals  have 
the  same  biases  about  juvenile  crime  and  delinquency  as  the 
general  population.  Because  of  ethnic,  gender,  and 
anti-mental  health  biases,  there  may  be  little  or  no  thought 
given  to  mental  illness  and  how  that  illness  might  have 
contributed  to  the  behaviors  that  have  been  exhibited  by  the 
youth.  Additionally,  many  of  the  individuals  making  the 
decision  to  detain  or  refer  the  youth  for  mental  health 
evaluation  have  numerous  organizational  pressures  and 
constraints  that  might  prevent  them  from  referring  the  youth 
for  mental  health  services  even  when  the  need  is  felt  to  exist 
(Mulvey  & Reppucci,  1984). 

To  date,  few  studies  have  examined  the  difference  in 
characteristics  between  first  time  and  repeat  offenders  with 
mental  illness.  In  this  study,  we  examined  demographic, 
clinical,  and  criminal  history  and  their  relationship  to 
whether  the  youth  was  a repeat  or  first  time  offender  in  a 
sample  of  youth  referred  for  mental  health  services  at  a 
juvenile  corrections  facility.  The  hypotheses  that  we 
explored  in  this  study  were:  (a)  Demographic  variables  are  a 
greater  predictor  of  repeat  offending  than  clinical  variables; 
and  (b)  youth  with  a history  of  violent  offenses  are  more 
likely  to  be  repeat  offenders  than  youth  with  a history  of 
non-violent  offenses. 

Setting 

The  San  Fernando  Valley  Juvenile  Hall  (SFVJH)  is  one  of 
three  juvenile  halls  that  serve  Los  Angeles  County, 
California.  The  juvenile  halls  each  house  both 
pre-adjudicated  as  well  as  post-adjudicated  youth.  The 
facility  has  a mental  health  treatment  center  located  on  site 
which  provides  mental  health  services  to  approximately  500 
youth  each  year.  The  entire  facility  has  an  average  daily 
census  of  600  youth,  and  approximately  3200  youth  are 
detained  annually. 
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The  youth  can  be  referred  for  mental  health  evaluation  by 
either  (a)  nursing  staff  upon  intake,  (b)  detention  staff 
members  who  work  directly  with  youth  in  one  of  1 0 
housing  units,  (c)  teachers  who  teach  at  the  school  located 
on  the  grounds  of  the  juvenile  hall,  (d)  the  probation  officer 
who  worked  with  the  youth  prior  to  detention,  or  (e)  the 
parents  who  could  contact  the  mental  health  staff  directly. 
There  is  no  formal  mental  health  evaluation  either  at  the 
time  of  intake  or  at  any  time  during  the  youth's  stay  in 
juvenile  hall. 

Subjects 

The  subject  population  (N  = 244)  was  assembled  from  244 
consecutive  referrals  during  a six  month  period  (January  1- 
June  30).  All  youth  were  between  12  and  1 8 years  of  age. 

Methods 

This  study  involved  reviewing  the  mental  health  and 
probation  records  of  the  youth.  The  probation  department's 
master  files  were  explored  for  data  on  all  youth  in  the 
juvenile  hall  during  the  period  of  the  study  (N  = 3,283). 

Only  aggregate  data  was  available  on  the  non-referred 
youth;  individual  level  data  was  not  available.  The  items 
available  from  the  probation  department  chart  review 
included  the  current  offense,  age  at  first  arrest,  and  the  total 
number  of  detentions.  The  items  that  were  available  from 
the  mental  health  department  record  review  included  the 
following:  (a)  the  current  age  of  the  youth;  (b)  history  of 
previous  mental  health  treatment;  (c)  drug  and  alcohol  use 
history;  (d)  mental  health  diagnoses;  and  (e)  reason  for 
mental  health  referral,  which  was  classified  as  either  an 
internalizing  or  externalizing  behavior  based  on  the 
narrative  provided  by  the  referrer. 

The  offense  precipitating  the  youth's  current  incarceration, 
or  "current  offense,"  was  categorized  by  the  level  of 
violence.  If  the  youth  engaged  in  a behavior  in  which 
physical  force  or  the  threat  of  force  was  used  against 
another  individual,  or  a weapon  was  used  in  the  commission 
of  the  crime,  the  youth  was  classified  as  being  a violent 
offender.  For  our  sample  of  youth,  violent  offenses 
consisted  of  murder,  attempted  murder,  rape  and  sexual 
assaults,  assault  with  a deadly  weapon,  assault  and  battery, 
and  robbery  (if  a weapon  was  used).  Recidivism  rate 
differences  in  gender,  race,  age,  mental  health  service  use, 
drug  use  internalizing  behavior,  years  of  delinquency,  and 
whether  the  current  offense  was  a violent  offense  were 
explored  using  a logistic  regression  model.  Chi-square  and 
t-tests  were  used  to  compare  the  total  juvenile  hall 
population  with  our  sample  of  youth. 

Results 

The  population  of  244  patients  included  144  (59%)  first 
time  offenders  and  100  (41%)  repeat  offenders.  Our  sample 
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time  offenders  and  100  (41%)  repeat  offenders.  Our  sample 
of  referred  youth  is  compared  with  the  total  population  of 
youth  in  Table  1.  The  referred  youth  represented  7.4%  of 
the  population  at  the  juvenile  hall.  Of  the  referred  youth, 
40%  had  previously  received  mental  health  treatment.  Those 
referred  for  mental  health  evaluation  differed  significantly 
from  youth  in  the  general  population  of  youth  in  juvenile 
hall.  Females,  non-Hispanic  youth,  and  violent  youth  were 
over-represented,  while  Hispanic  youth  and  first  time 
offenders  were  under-represented  in  our  sample. 

Among  the  referred  sample,  123  (50%)  of  the  youth 
reported  the  current  use  of  alcohol  and/or  drugs.  Among 
those  who  used  drugs,  90%  reported  current  use  of  two  or 
more  drugs.  The  drugs  most  often  identified  were  alcohol, 
marijuana,  crack  cocaine,  methamphetamine,  and 
hallucinogens. 

Table  2 shows  the  results  of  the  logistic  regression.  Among 
referred  youth,  the  juvenile  justice  histo^  variables  were 
more  associated  with  recidivism  than  clinical  or 
demographic  variables.  Age  at  first  offense  was  most 
strongly  associated  with  recidivism,  whereas  violent 
offenders  were  half  as  likely  to  be  repeat  offenders. 

Minority  youth  were  twice  as  likely  to  be  repeat  offenders 
than  non-minority  youth,  and  youth  referred  for 
internalizing  behaviors  were  twice  as  likely  to  be  repeat 
offenders.  The  previous  use  of  mental  health  services  had  no 
affect  on  recidivism. 

Discussion 

The  referred  youth  in  our  sample  differed  significantly  from 
youth  in  the  overall  population  at  the  juvenile  hall  in  almost 
all  categories  measured.  The  sample  included  fewer 
first-time  offenders  and  Hispanics;  it  included  more  females 
and  more  youth  with  a history  of  serious  violent  offenses.  It 
is  likely  that  these  findings  reflect  referral  bias;  the  juvenile 
hall  staff  may  have  been  more  likely  to  refer  certain  "types" 
of  youth  for  mental  health  services. 

Conversely,  the  results  of  this  study  suggest  that  the  arrest 
patterns  for  youth  referred  for  mental  health  services  are 
more  likely  to  be  affected  by  the  juvenile  justice  history  of 
the  youth  than  by  clinical  factors,  including  drug  use  and 
previous  mental  health  service  use.  The  number  of  years  of 
delinquency  appeared  to  be  the  strongest  predictor  of  repeat 
offending  within  the  group  of  referred  youth.  These  results 
are  consistent  with  those  which  show  early  age  of  onset  of 
offending  to  be  strongly  correlated  with  recidivism  in 
non-clinical  populations  (Lattimore,  1994;  Tolan,  1995). 
Thus,  the  treatment  modalities  that  are  used  with  juvenile 
offenders  who  have  emotional  disturbances  must  include 
treatments  which  focus  on  the  youths'  delinquent  behavior 
as  well  as  the  clinical  symptoms  which  are  non-criminal  in 
nature. 

There  was  a trend  that  violent  offenders  were  half  as  likely 
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to  be  repeat  offenders  as  non-violent  offenders,  although 
this  did  not  reach  statistical  significance  (p  = .05).  This 
finding  was  unexpected,  since  literature  supports  the  fact 
that  youth  tend  to  start  out  with  low  level  crimes  that 
escalate  to  more  serious  offenses,  and  therefore,  repeat 
offenders  would  be  more  likely  to  be  violent  offenders  than 
first  time  offenders.  The  youth  in  this  study  tended  to 
commit  a single  violent  offense  that  caused  them  to  be 
detained  for  a longer  period  of  time  compared  to  other 
studies  where  youth  committed  several  violent  offenses 
before  committing  a serious  violent  offense.  It  is  possible 
that  these  youth  differ  in  some  way  from  other  youth,  or 
they  are  perceived  by  staff  members  in  a way  that  would 
cause  them  to  have  different  referral  patterns  from  other 
detained  youth. 

This  study  has  several  limitations  including  a lack  of 
objective  clinical  information  on  each  of  the  youth  (e.g.,  the 
type  and  amount  of  previous  services  used,  family  history 
data,  and  information  on  staff  attitudes  about  the  youth 
which  could  significantly  alter  referral  patterns).  Despite 
these  limitations,  this  study  suggest  that  a component  of  the 
treatment  of  youth  with  mental  illness  who  are  at  risk  for 
detention  in  the  juvenile  justice  system  should  address 
violence  and  its  potential  impact  on  youth.  Additionally,  a 
special  effort  must  be  undertaken  to  determine  the  needs  of 
ethnic  minorities,  especially  Hispanics,  who  are  grossly 
underrepresented  among  the  referred  youth. 
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Introduction 


The  rapid  growth  of  mediation  in  fields  as  diverse  as 
divorce  mediation  and  commercial  dispute  resolution  has 
raised  the  interest  of  a growing  number  of  child  welfare 
service  providers  and  scholars  interested  in  finding  a more 
constructive  process  for  developing  collaborative  service 
plans  with  families  and  youth  (Mayer,  1987).  Several 
jurisdictions  have  already  introduced  mediators  at  various 
levels  of  the  child  protection  system  (Barsky,  1995a; 
Thoennes,  1994;  Savoury,  Beals,  & Parks,  1995).  Mediation 
is  proposed  by  some  advocates  as  a more  effective  method 
for  engaging  families  in  treatment  (Bernstein,  Campbell,  & 
Sookraj,  1993).  Others  argue  that  it  could  produce  major 
time  and  cost  savings  by  avoiding  protracted  court  hearings 
(Morden,  1989).  Yet  others  maintain  that  mediation 
provides  a basis  for  empowering  families  and  ensuring  that 
child  protection  services  proceed  in  a manner  that  is  least 
intrusive  to  families  and  children  (Regehr,  1994; 

Wildgoose,  1987). 

Most  writings  to  date  have  focused  either  on  developing  a 
rationale  for  using  mediation  in  child  protection  (Eddy, 

1992;  Palmer,  1989)  or  on  evaluation  of  various  programs 
(Campbell  & Rodenburgh,  1994;  Centre  for  Policy 
Research,  1992;  Wildgoose  & Maresca,  1994).  However, 
there  has  been  insufficient  work  describing  the  process  of 
child  protection  (CP)  mediation,  and  analyzing  ways  in 
which  mediation  can  be  integrated  into  child  protection 
services.  Accordingly,  this  study  was  designed  to  provide  an 
in  depth  understanding  of  the  process  of  mediation  in  child 
protection,  and  to  identify  the  essential  aspects  which  may 
contribute  to  developing  more  effective  working 
relationships  with  child  welfare  families.  This  study  focuses 
on  neutrality,  one  of  the  primary  aspects  of  CP  mediation 
identified  by  the  research  participants.  This  study  is  distinct 
from  prior  studies  because  it  entails  the  use  of  extensive 
interviews  with  family  members,  as  well  as  with 
professionals  involved  in  mediation  and  other  CP  processes. 
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An  analysis  of  their  experiences  contributes  to  the  child 
welfare  and  mediation  literature:  (a)  by  beginning  to 
determine  the  critical  experiences  of  parties  involved  in  the 
different  processes,  (b)  by  developing  a framework 
delineating  the  essential  aspects  of  the  mediation  process 
that  are  unique  to  mediation,  and  (c)  by  identifying 
mediative  skills  and  strategies  that  can  be  employed  in  other 
areas  of  child  protection  services. 

Methods 

In  order  to  develop  a better  understanding  of  the  dynamics 
of  child  protection  mediation,  this  study  used  naturalistic 
inquiry  methods  (Denzin  & Lincoln,  1994).  The  author 
conducted  a series  of  exploratory  "long  interviews" 
(McCracken,  1988)  with  17  adult  family  members, 
mediators,  and  child  protection  workers  (CPWs)  who  had 
been  directly  involved  in  five  mediation  cases.  The  primary 
strategy  of  these  interviews  was  to  have  research 
participants  discuss  what  they  felt  were  the  critical  issues  in 
their  experiences  with  mediation.  Six  pre-mediation 
interviews,  and  six  additional  interviews  were  also 
conducted  with  family  members  and  CPWs  from  three 
non-mediation  comparison  cases.  The  author  asked 
participants  in  these  interviews  to  describe  their  experiences 
with  CP  interventions,  including  investigations,  social 
casework,  case  conferencing,  solution  focused  therapy,  and 
child  protection  court.  The  experiences  of  the  research 
participants  in  the  pre-mediation  interviews  and  in  the 
non-mediation  cases  provided  the  contrast  points  used  in 
identifying  the  essential  aspects  of  mediation  (Spradley, 
1979). 

Mediation  cases  were  sampled  on  the  basis  of  availability 
and  willingness  to  participate  from  a pool  of  approximately 
30  mediation  cases  seen  by  the  Center  for  Child  and  Family 
Mediation  in  Toronto.  The  Centre  offers  mediation  to 
Toronto  families  who  are  involved  in  the  child  protection 
system.  In  terms  of  demographics  of  research  participants, 
most  families  involved  in  both  the  mediation  and 
non-mediation  cases  were  headed  by  a single  mother  who 
was  either  receiving  government  assistance  or  earning  less 
than  $30,000.00  annually.  All  but  one  family  had  either  one 
or  two  children.  These  demographics  are  consistent  with  the 
general  population  of  families  involved  in  child  protection 
mediation  in  Toronto. 

The  sample  of  non-mediated  cases  was  dravm  purposefully 
on  a case-by-case  basis  by  having  CPWs  from  the  mediation 
sample  identify  matching  cases  from  their  own  caseload. 
Selection  was  done  to  ensure  that  the  comparison  cases 
matched  the  mediation  cases  in  terms  of  demographic 
profiles,  type  and  severity  of  maltreatment,  placement 
status,  and  level  of  involvement  by  participants  in  the  case 
plaiming  process.  In  terms  of  ethnicity,  families  in  the 
mediation  sample  included  people  from  European,  African, 
and  mixed  Euro-Affican  backgrounds.  All  of  the  families  in 
the  non-mediation  sample  came  from  Euro-Canadian 
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the  non-mediation  sample  came  from  Euro-Canadian 
backgrounds.  Matching  for  ethnicity  was  not  prioritized, 
because  both  mediators  and  child  protection  workers 
suggested  that  the  most  important  factor  to  consider  was  the 
type  of  intervention  used  in  the  non-mediation  cases. 
Therefore,  the  investigator  tried  to  select  comparison  cases 
in  which  mediation-like  interventions  were  employed. 

The  interviews  were  audio-taped  and  transcribed.  In  a 
situation  where  a parent  asked  not  to  be  taped,  written  notes 
were  used  to  develop  a transcript.  Each  interview  was  read, 
re-read  and  analyzed  as  it  was  completed,  rather  than 
waiting  to  analyze  all  of  the  interviews  together.  Key 
themes  and  patterns  from  the  interviewees'  perspectives 
were  identified  from  the  transcripts  (Lincoln  & Cuba,  1985; 
Taylor  & Bogden,  1984).  The  interview  guides  were  altered 
for  subsequent  interviews  in  order  to  further  explore 
important  topics  and  perspectives  raised  by  interviewees 
that  had  not  originally  been  identified  by  the  researcher. 
McCracken's  five-stage  model  for  data  analysis  was 
employed  (McCracken,  1988).  The  transcribed  texts  were 
processed  for  analysis  with  the  Ethnograph  (Seidel  & Clark, 
1984)  computer  software  package.  Lincoln  & Cuba's  (1985) 
method  for  establishing  trustworthiness  was  used 
throughout  the  study.  "Credibility,"  "transferability," 
"dependability,"  and  "confirmability"  (the  qualitative 
research  equivalents  of  internal  validity,  external  validity, 
reliability  and  objectivity)  were  each  considered. 

Results 

During  the  initial  stage  of  analysis,  the  researcher  attempted 
to  identify  as  many  themes  as  the  participants'  data 
indicated.  This  effort  resulted  in  a broad  base  of  92  themes. 
Many  of  these  themes  overlapped,  and  some  of  them  did  not 
prove  to  be  significant  in  later  analysis.  However,  since  the 
purpose  of  the  study  was  to  look  at  mediation  from  the 
participants'  perspectives,  it  was  beneficial  to  start  with  a 
broad  identification  of  themes.  The  initial  themes  were 
loosely  categorized  into  nine  pools  (Barsky,  1995a):  1) 
Intentions  and  Concerns  of  the  Parties;  2)  Mediation 
Interventions;  3)  Dynamics  between  the  Parties;  4) 
Appropriateness  for  Mediation;  5)  Comparisons  made  by 
! the  Parties;  6)  Roles  of  the  Parties;  7)  Factors  Affecting 
Decisions;  8)  Outcomes;  and  9)  Impact  of  the  Research. 

In  the  next  stage  of  analysis,  ten  essential  elements  of  the 
mediation  process  were  identified.  These  elements  were 
derived  in  part  from  the  aspects  of  mediation  that  mediation 
participants  identified  as  being  most  important.  There  were 
also  certain  recurring  elements  that  the  parties  did  not 
specifically  identify  as  most  important,  but  which  emerged 
implicitly  as  defining  features  of  their  experiences  in  the 
mediation  process.  One  of  the  difficulties  in  this  process 
was  isolating  elements  that  the  interviewer  may  have 
suggested  through  the  use  of  leading  questions.  For  this 
purpose,  it  was  useful  to  look  at  the  early  portions  of  the 
interview  where  the  interviewer  asked  more  general 
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interview  where  the  interviewer  asked  more  general 
questions.  Eventually,  ten  distinct  essential  elements  were 
identified:  1)  alliance,  2)  bringing  parties  together,  3) 
facilitating  communication,  4)  keeping  peace,  5)  developing 
options,  6)  enhancing  understanding,  7)  focusing  the  parties, 
8)  contracting,  9)  neutrality,  and  10)  fairness.  Comparisons 
with  non-mediation  cases  suggested  that  many  of  these 
elements,  though  essential  to  mediation,  were  not 
necessarily  unique.  CPWs  involved  in  case  conferencing 
and  solution-focused  intervention  utilized  techniques  similar 
to  those  used  by  mediators.  However,  mediator  neutrality 
was  found  to  be  unique  to  CP  mediation. 

Discussion 

Child  protection  mediation  is  a peculiar  use  of  mediation, 
given  that  one  of  the  clients  is  a professional  who 
traditionally  assumes  a mediation  role  in  his/her  own  work. 
When  suspicions  of  child  abuse  or  neglect  are  brought  to  the 
attention  of  a child  protection  agency,  the  assigned  worker 
is  mandated  to  investigate  and  intervene  in  order  to  ensure 
that  the  child  is  not  at  risk.  In  the  context  of  child 
protection,  a worker  could  mediate  between  disputing 
family  members  about  how  they  will  share  responsibility  for 
a child's  needs.  Similarly,  a worker  could  mediate  between 
parents  and  foster  parents  about  how  and  when  the  parents 
will  spend  time  with  the  child.  But  is  a child  protection 
worker  truly  able  to  function  as  a neutral  third  party?  For 
CP  mediation  to  have  value  as  a distinct  intervention,  it 
needs  to  be  defined  in  a way  that  is  different  from  just 
"good  clinical  practice"  by  CPWs.  Otherwise,  a CPW  could 
fulfill  the  role,  making  the  use  of  an  independent  mediator 
redundant  (Barsky,  1995b).  One  of  the  major  findings  from 
the  present  research  was  that  neutrality  does  differentiate  the 
role  of  a mediator  from  the  role  of  a CPW. 

Although  the  present  research  was  not  intended  to  evaluate 
the  effectiveness  of  CP  mediation,  research  participants 
tended  to  volunteer  positive  feedback  on  their  experiences 
in  mediation.  In  particular,  informants  reported  that 
mediation  helped  them  to  resolve  issues  without  going  to 
court,  helped  build  a more  positive  relationship  between 
family  members  and  the  agency,  and  gave  all  parties  a sense 
of  being  heard  and  treated  fairly.  CP  mediation  is  not  a 
panacea,  as  noted  by  the  frustration  of  some  parties  in  cases 
that  did  not  settle.  However,  given  the  efficacy  of  mediation 
in  this  small  sample  of  cases,  further  study  is  warranted. 
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Introduction 

Many  studies  have  shown  that  only  a small  proportion  of 
youth  with  psychiatric  problems  receive  professional  help 
(Bums,  1991),  and  that  youth  with  depressive  or  anxiety 
disorder  are  less  likely  to  be  referred  for  services  than  youth 
with  dismptive  disorder  (Anderson,  Williams,  McGee,  & 
Silva,  1987;  Cohen,  Kasen,  Brook,  & Struening,  1991). 
However,  few  studies  have  examined  the  service  use 
patterns  of  children  with  depressive  disorder  in  community 
samples.  This  study  addresses  two  questions: 

1.  How.likely  are. children  with  depressive  disorders 
to  use  mental  health  services? 

2.  Are  the  patterns  of  service  use  and  parents' 
perception  of  service  use  barriers  the  same  for 
children  with  depressive  disorder  as  for  those  with 
dismptive  disorder? 
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Method 


Sample  and  Data 

Data  were  derived  from  a multi-site  community  survey 
conducted  in  1992  and  funded  by  NIMH:  the  Methods  for 
the  Epidemiology  of  Child  and  Adolescent  Mental 
Disorders  (MECA)  Study  (Lahey  et  al,  in  press).  This 
probability  sample  consists  of  1285  child  (ages  9-17)  and 
parent/guardian  pairs,  and  was  drawn  from  four  geographic 
areas:  (a)  Hamden,  East  Haven  and  West  Haven, 

Connecticut  (N  = 314);  (b)  DeKalb,  Rockdale,  and  Henry 
Counties,  Georgia  (N  = 299);  (c)  Westchester  County,  New 
York  (N  = 360);  and  (d)  San  Juan,  Puerto  Rico  (N  = 3 12). 

Structured,  in-person  interviews  were  conducted  to  obtain 
psychiatric  diagnoses,  as  measured  by  NIMH's  Diagnostic 
Interview  Schedule  for  Children  (DISC),  version  2.3 
(Shaffer,  et  al.,  in  press),  and  impairment,  measured  by  both 
the  Non-Clinician  Global  Assessment  Scale  for  Children 
(Bird,  submitted)  and  the  Columbia  Impairment  Scale  (Bird 
et  al.,  1993).  The  Service  Utilization  and  Risk  Factors 
(SURF)  interview  was  also  administered  (Leaf  et  al.,  in 
press). 

Measures  of  Depressive  and  Disruptive  Disorders 

• Depressive  disorder  (parent  or  child  report):  Meets 
DISC  criteria  for  major  depressive  disorder  and/or 
dysthymic  disorder,  is  positive  on  diagnosis-specific 
impairment  (as  measured  by  the  DISC),  and  has  a 
CGAS  score  (interviewer  assessment)  lower  than  or 
equal  to  70. 

• Disruptive  disorder  (parent  or  child  report):  Meets 
DISC  criteria  for  attention  deficit  hyperactivity 
disorder,  oppositional  defiant  disorder,  or  conduct 
disorder,  is  positive  on  diagnosis-specific  impairment 
(as  measured  by  the  DISC),  and  has  a CGAS  score 
(interviewer  assessment)  lower  than  or  equal  to  70. 

Measures  of  Service  Utilization  and  Barriers  to  Services 

• Ever  used  any  service  (parent  or  child  report):  Child 
has  ever  used  any  of  the  following  services  for 
emotional  or  behavior  problems,  or  for  the  use  of 
alcohol  or  drugs  (Leaf  et  al,  in  press):  (1)  mental 
health  specialist  (see  below  for  details);  (2)  medical 
professionals;  (3)  school-based  services;  (4)  social 
services;  (5)  clergy;  and  (6)  other  (e.g.,  spiritualists, 
herbalists). 

• Ever  used  any  mental  health  specialist  (parent 
report):  Child  has  ever  seen  any  of  the  following:  (1) 
psychiatrist,  (2)  psychologist,  (3)  counselor,  or  (4) 
social  workers. 

• Ever  used  two  or  more  non-school  mental  health 
services  (parent  report):  Ever  used  two  or  more  types 
of  non-school  mental  health  services  with  one  of  them 
being  a mental  health  specialist. 
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being  a mental  health  specialist. 

• Parentis  perceived  barriers:  Barriers  to  the  use  of 
child  mental  health  services  are  measured  by  parental 
report  of  the  number  of  barriers  to  service,  based  on 
17  questions  relating  to  the  cost  of  treatment,  access 
to  services,  and  knowledge  of  and  attitudes  towards 
mental  health  services. 

Results 

In  the  MECA  sample,  176  children  (13.7%)  met  diagnostic 
criteria  for  either  depressive  or  disruptive  disorders  (Shaffer, 
et  al.,  in  press).  Among  them,  44  children  were  diagnosed 
with  depressive  disorder  only,  96  children  with  disruptive 
behaviors  only,  and  36  children  met  the  criteria  for  both 
depressive  and  disruptive  disorders. 

Service  use  patterns  were  first  compared  for  children  with 
depressive  disorder  only,  disruptive  disorder  only,  both 
disorders,  and  neither  of  the  two  disorders.  To  further  test 
the  hypothesis  that  children  with  depressive  disorder  use 
less  service  than  children  with  disruptive  disorder,  the 
disruptive  disorder  only  group  and  the  group  with  both 
disorders  were  combined  into  a "Disruptive  disorder"  group. 

Children  with  either  one  or  both  disorders  used  more 
services  than  those  with  neither  of  the  disorders.  For 
children  with  depressive  disorder,  61.4%  have  used  any 
service;  only  34.1%  have  ever  seen  a mental  health 
specialist,  and  only  25.0%  have  ever  used  2 or  more 
non-school  mental  health  services.  Children  with  disruptive 
disorder,  regardless  of  having  depressive  disorder  or  not,  are 
more  likely  to  use  service  than  children  with  depressive 
disorder  only.  The  difference  is  statistically  significant  for 
seeing  a mental  health  specialist  (see  Table  1). 

Three  barriers  were  found  to  differ  significantly  between 
children  with  depressive  disorder  only  and  children  in  other 
groups.  More  parents  of  children  with  depressive  disorder 
only  reported  that  mental  health  professionals  were 
unfriendly  and  that  they  did  not  trust  mental  health 
professionals  than  did  parents  of  children  with  disruptive 
disorders  or  neither  of  the  two  disorders.  Also,  more  parents 
of  children  with  depressive  disorder  only  reported  that  they 
were  unable  to  get  an  appointment  (see  Table  2). 

Logistic  regression  analyses  of  lifetime  service  use  were 
conducted  for  the  total  sample  in  five  steps  (5  models).  In 
each  model,  study  site,  mother's  education,  family  income, 
health  insurance,  parental  psychopathology,  child's  age,  sex, 
ethnicity,  and  number  of  lifetime  chronic  illnesses  were 
controlled.  In  addition  to  these  control  variables,  depressive 
disorder  was  entered  in  Model  1;  disruptive  disorder  was 
entered  in  Model  2;  both  depressive  and  disruptive  disorders 
were  entered  in  Model  3.  Model  4 included  all  the  variables 
in  Model  3 as  well  as  parent  and  child  ratings  on  the 
Columbia  Impairment  Scale  (i.e.,  CIS  score  of  16  or  more). 
Model  5 included  all  the  variables  in  Model  4 as  well  as 
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Model  5 included  all  the  variables  in  Model  4 as  well  as 
parent  perception  of  child  service  need. 

When  depressive  disorder  and  disruptive  disorder  were 
entered  into  the  logistic  regression  separately  to  predict 
lifetime  service  use,  the  Adjusted  Odds  Ratios  (AOR)  for 
disruptive  disorder  (Model  2)  for  all  three  service  use 
variables  were  much  higher  than  those  for  depressive 
disorder  (Model  1).  This  pattern  also  held  when  the 
depressive  and  disruptive  disorders  were  entered  into  the 
equation  simultaneously  (Model  3).  When  the  parent  and 
child  ratings  of  child  impairment  (CIS)  and  parent  perceived 
need  were  entered  into  the  equation  (Model  4 and  5),  the 
effects  of  depressive  disorder  became  non-significant,  while 
the  effects  of  disruptive  disorder  still  held  (see  Table  3). 

Discussion 

Consistent  with  other  studies  (Anderson  et  al,  1987;  Cohen 
et  al,  1991),  the  findings  here  show  that  children  with 
depressive  disorder  are  less  likely  to  use  services  than 
children  with  disruptive  disorder.  Also,  parents  of  children 
with  depressive  disorder  report  more  barriers  to  service  than 
children  with  disruptive  disorder.  These  findings  suggest 
that  children  with  depressive  disorders  are  less  likely  to  be 
identified  or  referred  to  mental  health  services,  and  that  they 
have  more  difficulties  in  accessing  services. 

Parent's  perception  of  the  child's  impairment  and  need  for 
service  use  plays  an  important  role  in  the  child  obtaining 
mental  health  services.  This  finding  may  be  related  to  the 
lack  of  service  use  by  children  with  depressive  disorder,  as 
this  and  other  internalizing  disorders  are  less  likely  to  be 
recognized  by  other  people,  including  the  parents. 
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Introduction 


Best  Practices  (Best  Practices  for  Educating  Students  with 
Serious  Emotional  Disturbance  in  Their  Local  Public 
Schools  and  Communities)  was  a three  year  grant  funded  by 
the  US  Office  of  Education  and  Rehabilitative  Services.  The 
purpose  of  the  grant  was  to  use  best  practices  in  the 
development  of  programs  and  services  for  students  with 
serious  emotional  disturbance  (SED)  and  their  families,  so 
that  the  students  could  remain  at  their  local  public  schools, 
or  plans  could  be  developed  so  that  the  students  could  return 
from  alternative  placements.  The  grant  was  a collaborative 
effort  between  local  Vermont  schools,  the  University 
Affiliated  Program  of  Vermont  (UAP),  the  Vermont 
Interdisciplinary  Team  for  Intensive  Special  Education 
(State  I-Team),  and  the  Vermont  State  Department  of 
Education,  Division  of  Instructional  Support  Services. 
Mechanisms  with  local  and  state  Departments  of  Education, 
Mental  Health,  and  Social  Services  were  developed  to 
facilitate  the  model  programs. 


I The  model  programs  and  support  services  that  were 

I developed  through  involvement  of  the  Best  Practices  project 

||  focused  on  parent  and  student  centered  services, 

I family-school  collaboration,  collaborative  teaming, 

I classroom  accommodations,  peer  support,  instructional 

I support  services,  teaching  prosocial  skills,  interagency 

a involvement,  behavior  management,  and  transition 

i planning. 
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Method 

Subjects  and  Sites 

A total  of  twenty-seven  schools  in  twenty-two  different 
school  districts  across  Vermont  participated  as  sites  in  this 
project.  All  the  sites  had  students  with  serious  emotional 
disturbance  as  defined  by  the  Federal  Educational  definition 
of  severe  disabilities  and  serious  emotional  disturbance 
(P.L.  100-476  of  1990),  or  Vermont's  Act  264  definition  of 
severe  emotional  disturbance  (Vermont  Act  264  of  1988). 

The  school  sites  chosen  for  participation  in  the  project  met 
the  following  criteria:  (a)  administrative  commitment  at  the 
building  level  for  educating  students  with  serious  emotional 
disturbance  within  the  regular  education  environment;  (b) 
willingness  to  establish  a "school-wide  planning"  process  to 
assist  in  developing  plans  for  implementing  best  practices 
for  all  students  (Fox  & Williams,  1991); 

(c)  on-site  case  manager  for  each  student  with  serious 
emotional  disturbance  involved  in  the  project;  (d) 
commitment  from  school  administration  and  school  staff  to 
in-service  training  and  participation  in  the  project;  (e) 
commitment  to  provide  school  based  family  support 
services  and  to  involve  parents  in  all  aspects  of  students' 
programs;  (f)  commitment  to  communicate  with  and  involve 
local  mental  health  and  social  service  agencies  to  support 
each  student's  program  needs;  and  (g)  willingness  to  use  an 
individual  student  support  team  process  (Hamilton, 
Welkowitz,  Mandeville,  Prue,  & Fox,  1995)  to  assist  in 
developing  and  implementing  educational  activities  for  at 
least  one  student  with  or  at  risk  of  serious  emotional 
disturbance. 

The  thirty-six  participating  students  with  SED,  in  grades  K 
through  12,  were  either  at  risk  of  removal  from  their 
educational  setting  or  were  returning  to  their  regular 
educational  setting  from  a more  restrictive  placement. 

Intervention 

An  individualized  approach  to  each  student  and  school 
participating  in  this  project  was  emphasized.  A school 
planning  team  was  formed  at  each  school  site.  The  school 
planning  teams  involved  administrators,  school  board 
members,  parents,  regular  educators,  and  special  educators, 
a member  of  the  school's  instructional  support  team,  other 
community  mental  health  and  social  service  agencies,  and 
local  community  members.  One  focus  of  the  school 
planning  team  was  to  examine  existing  educational  practices 
and  to  recommend  changes  and  improvements  which  would 
enhance  the  education  of  all  students.  Additional  activities 
included:  (a)  identifying  resources  which  were  available  to 
the  school;  (b)  determining  the  availability  of  best  practices 
for  students;  (c)  identifying  support  needs  of  students, 
families,  and  staff;  (d)  identifying  needed  changes  in  school 
policy  and  service  related  delivery;  and  (e)  developing  an 
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policy  and  service  related  delivery;  and  (e)  developing  an 
action  plan  for  attainment  of  one  goal  as  determined  by  the 
school  planning  team. 

Individual  student  planning  teams  were  formed  around  each 
child  and  their  family.  Members  of  these  teams  attended  a 
Fall  Institute  which  focused  on  using  a functional 
assessment  process  (Hamilton  et  al.,  1995)  to  determine 
areas  of  need  for  the  student,  identifying  and  making 
educational  accommodations,  and  using  a problem  solving 
process  to  identify  prevention,  teaching,  and  response 
strategies  for  problematic  behaviors  by  the  student.  Each 
individual  student  planning  team  typically  included  the 
student's  parent(s),  classroom  teacher,  special  educator, 
guidance  counselor,  a member  of  the  school's  instructional 
support  team,  and  the  school  principal.  Extended  Team 
members  included  the  State  I-Team  regional  consultant, 
representatives  from  mental  health,  social  services,  and 
other  relevant  community  agencies. 

The  purpose  of  the  student's  planning  team  was  to  support 
and  assist  the  teacher  in  developing  and  implementing  the 
student's  educational  activities,  to  provide  a vehicle  for 
home-school  communication  and  support,  and  to  develop 
transition  plans  between  grade  levels,  schools,  and  into 
adulthood.  Typical  activities  of  the  student  planning  team 
involved:  (a)  identifying  training  and  information  needs;  (b) 
developing  long-range  educational  plans  for  the  student;  (c) 
adapting  and  accommodating  regular  education  activities; 

(d)  problem  solving  and  implementing  strategies  for 
problematic  behavior;  and  (e)  accessing  mental  health  and 
social  services  agencies  for  needed  services. 

Evaluation 

The  project  used  quantitative  and  qualitative  methodology 
to  evaluate  outcomes.  Semi-structured  interviews  were 
conducted  at  the  beginning  and  end  of  each  school  year  with 
the  students  with  SED,  their  peers,  their  parents,  and  their 
regular  and  special  education  teachers. 

Surveys  and  questionnaires  were  completed  at  the  beginning 
and  end  of  each  school  year  by  the  student's  family,  his  or 
her  peers'  families,  the  student's  administrator,  and  regular 
and  special  educator.  Members  of  the  students'  individual 
support  teams  completed  a survey  in  regard  to  their  teaming 
process  and  their  success  in  addressing  the  needs  of  students 
with  SED.  T-test  analyses  were  conducted  for  each  question 
to  determine  whether  the  participant's  response  prior  to 
participation  in  the  project  was  significantly  different  from 
their  response  following  participation,  support,  and  training. 

A behavior  checklist  was  also  completed  at  the  beginning  of 
the  school  year  and  at  the  end  of  the  school  year  by:  (a)  the 
student,  if  eleven  years  old  or  older,  using  the  Youth  Self 
Report  (YSR;  Achenbach,  1991b);  (b)  his  or  her  teachers, 
using  the  Teacher's  Report  Form  (TRF;  Achenbach,  1991a); 
and  (c)  his  or  her  parents,  using  the  Child  Behavior 
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and  (c)  his  or  her  parents,  using  the  Child  Behavior 
Checklist  (CBCL;  Achenbach,  1991).  T-test  analyses  were 
conducted  to  determine  whether  the  scores  were 
significantly  different  for  each  of  these  measures  following 
training,  support,  and  participation  in  the  project. 

Results 

This  project  involved  thirty-six  students  with  SED  who 
were  in  grades  K through  12  in  twenty-seven  schools  in 
twenty-two  different  school  districts,  their  respective 
families,  educational  and  related  service  providers, 
community  based  service  providers,  and  communities.  All 
but  two  of  these  students  were  successfully  educated  within 
the  regular  education  classroom  in  their  local  public  school. 

Surveys  and  Checklists 

There  were  no  statistically  significant  differences  in  how  the 
classmates'  families  (N  = 174)  responded  to  the  Family 
Survey  questions  (see  Table  IT  or  for  the  administrators' 
responses  (N  = 13)  on  the  Admin-istrator's  Survey  (see 
Table  21.  between  the  first  and  second  administration. 
Overall,  their  responses  were  neutral  in  regard  to  any  of  the 
given  statements. 

There  were  several  statistically  significant  differences  for 
the  teachers'  responses  (N  = 30)  between  the  first  and 
second  administration  (see  Table  31  of  the  General 
Education  Teacher  Survey  and  Special  Education  Teacher 
Survey,  and  in  how  team  members  (N  = 118)  responded 
between  administrations  of  the  Individual  Student  Planning 
Team  Survey  (see  Table  41. 

There  were  no  statistically  significant  differences  between 
the  pre-  and  post-administrations  of  the  CBCL,  YSR,  and 
TRF.  The  students  with  SED  continued  to  be  perceived  by 
others  and  themselves  as  having  behaviors  within  the 
clinical  range  following  participation  in  the  project. 

Semi-Structured  Interviews 

The  students  did  not  specifically  attribute  their  experiences 
to  being  in  inclusive  classrooms;  however,  three  themes  did 
emerge  from  the  data  that  were  related  to  going  to  school 
with  typical  peers.  These  included  how  students  solve 
problems,  how  students  cope  with  challenging  situations, 
and  a student's  desire  to  help  others. 

All  of  the  educators  were  asked  what  effect  inclusion  had  on 
them  professionally  and  personally.  Their  responses  focused 
on:  (a)  quality  of  education  for  the  student;  (b)  awareness  of 
individual  student  differences  and  needs;  (c)  awareness  of 
the  needs  of  all  students;  (d)  patience,  workload  and  stress; 
and  (e)  style  of  teaching.  When  the  responses  from  the  two 
interview  sessions  with  each  teacher  were  compared,  it  was 
found  that  the  educators'  perceptions  did  not  change.  For 
example,  if  their  perceptions  were  initially  negative  in 
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example,  if  their  perceptions  were  initially  negative  in 
regard  to  inclusion  of  students  with  serious  emotional 
disturbance  in  the  regular  classroom,  they  remained 
negative,  despite  the  student's  program  being  successful 
overall. 

With  the  exception  of  one  participant,  the  special  educators 
in  this  study  felt  that  inclusion  had  made  a positive  impact 
on  their  professional  and  personal  lives.  One  of  the  primary 
benefits  was  that  the  educators  felt  they  were  able  to  work 
as  a team  to  serve  these  students. 

With  respect  to  parents'  involvement  in  the  team  process, 
the  results  indicated  that  parents  varied  in  the  degree  to 
which  they  felt  like  a full  team  member  with  an  equal  voice 
in  decision-making.  On  one  end  of  the  continuum  were 
parents  who  expressed  minimal  involvement  with  the  teams, 
and  at  the  other  extreme  were  parents  who  indicated  that 
they  were  "most  definitely"  full  team  members,  with  equal 
say  in  all  decisions  regarding  their  children. 

Discussion 

Students  with  SED  are  often  viewed  as  the  one  group  of 
students  receiving  special  education  services  who  cannot  be 
successfully  included  within  the  regular  classroom 
environment  in  their  local  public  schools.  Few  studies, 
however,  have  been  done  in  schools  which  fully  include 
these  students  in  the  regular  classroom. 

In  the  Best  Practices  Project,  the  development  and 
implementation  of  educational  strategies  which  emphasized 
prevention  and  the  teaching  of  replacement  skills  allowed 
almost  all  of  the  participating  students  to  continue  to  receive 
their  educational  services  within  the  regular  education 
classrooms.  The  findings  from  the  project  evaluation 
suggest  that  with  adequate  school-wide  planning  and  access 
to  training,  and  use  of  an  individual  student  support 
planning  team  process,  peers,  parents,  administrators, 
special  educators,  and  regular  educators  are  willing  to  have 
students  with  SED  within  the  regular  classroom  in  their 
local  public  school. 

Based  on  these  initial  findings,  the  above  study  is  now  being 
extended  to  include  up  to  forty  rural  school  sites  across  the 
nation.  It  is  expected  that  participation  as  a site  will  lead  to 
supportive  and  proactive  interagency  planning  and  inclusion 
of  children  and  youth  with  SED  in  their  general  education 
classroom  in  their  local  public  school  in  other  rural  states. 
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Onward  to  Outcomes  in  the  School  of  the 
Future 


Introduction 

In  conjunction  with  the  school  reform  and  children's  mental 
health  movements,  the  concept  of  full  service  schools  has 
received  considerable  attention  in  the  last  few  years 
(Dryfoos,  1994).  At  the  same  time,  increasing  attention  has 
been  given  to  the  need  for  outcome  measures  of 
effectiveness  across  the  spectrum  of  health  and  human 
services.  This  summary  reports  on  efforts  to  conduct 
longitudinal  research  to  assess  the  efficacy  of  a 
comprehensive  school-based  service  progr^  in  Texas,  the 
School  of  the  Future  project. 

The  School  of  the  Future,  funded  by  the  Hogg  Foundation 
for  Mental  Health,  was  a five  year  demonstration  project  in 
four  Texas  cities:  Austin,  Dallas,  Houston,  and  San 
Antonio.  Each  site  received  a grant  of  $50,000  per  year  for 
five  years  to  develop  and  coordinate  health  and  social 
services  through  targeted  schools  in  low  income, 
predominantly  ethnic  minority  locales.  Services  were 
provided  for  children  from  pre-natal  care  through  age  15, 
and  their  families.  The  grant  essentially  provided  salary  and 
benefits  for  one  full  time  position,  the  Project  Coordinator, 
with  a small  operating  budget.  The  grant  was  intentionally 
small  enough  that  direct  purchase  of  services  was  a very 
limited  option,  thus  directing  efforts  of  the  Coordinators 
toward  collaborative  efforts  with  existing  resources,  or 
securing  outside  funding  for  services.  The  purpose  was  to 
develop  a potentially  replicable,  affordable  model  of 
comprehensive  service  delivery  which  could  be  used  in 
localities  with  limited  financial  resources. 

In  addition  to  the  one  million  dollars  in  direct  grant  support, 
the  Hogg  Foundation  committed  an  equal  amount  to  indirect 
support,  primarily  for  evaluation  of  the  project.  The 
evaluation  plan  for  the  School  of  the  Future  was  enhanced 
through  consultation  with  directors  of  two  earlier  projects. 
Dr.  James  Comer  (Comer,  1980;  1988)  and  Dr.  Edward 
Zigler  (Zigler,  1989),  and  a school-based  service  project  in 
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Philadelphia  funded  by  the  Center  for  Education  in  the  Inner 
Cities  (CEIC)  Project  at  Temple  University  (Keir,  Culler,  & 
Diamond,  1993).  An  interdisciplinary  team  of  evaluation 
research  consultants  was  drawn  from  the  University  of 
Texas  at  Austin,  the  University  of  Texas  Health  Science 
Center  in  Houston,  the  University  of  Texas  Health  Science 
Center  in  San  Antonio,  and  Southwestern  Medical  School  in 
Dallas. 

The  purpose  of  the  School  of  the  Future  evaluation  was  (a) 
to  assist  in  site  planning  for  program  development,  (b)  to 
inform  key  decision  m^ers  including  legislators,  and  (c)  to 
develop  a blueprint  for  replication  of  the  model. 
Additionally,  it  was  recognized  that  the  development  of  a 
large  scale  longitudinal  database  of  the  physical  and  mental 
health  of  primarily  Hispanic  children  from  low  income 
neighborhoods  was  unique  and  could  foster  additional 
on-going  research.  A total  of  approximately  12,000  students 
across  the  four  program  sites  were  included  in  the 
longitudinal  database.  This  report  focuses  on  one  site  (Site 
A)  with  about  3,000  students. 

The  intervention  model  represented  in  the  School  of  the 
Future  was  based  on  an  implicit  theory  of  practice  that  a 
program  of  comprehensive  integrated  family  social  services, 
made  accessible  through  local  schools,  will  improve  family 
functioning,  student  health,  mental  health,  and  academic 
performance.  The  research  question  which  drove  the 
evaluation  efforts  was:  Given  a shared  vision  of 
comprehensive  services  and  community  empowerment,  can 
a targeted  investment  in  a single  individual  serve  as  a 
sufficient  catalyst  to  identify,  implement,  and  coordinate 
services  to  improve  student  mental  and  physical  health,  and 
academic  performance? 
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Method 


Evaluation  Design 

The  evaluation  included  two  primary  components,  a 
systemic  analysis  and  an  effectiveness  study  based  on 
student  outcomes.  Although  a number  of  students  did 
receive  direct  services  as  a result  of  the  School  of  the 
Future,  the  intervention  was  primarily  a systemic 
intervention&endash;the  introduction  of  a single  individual 
to  serve  as  a catalyst  for  the  development  and  delivery  of 
comprehensive  services.  The  systemic  analysis  was  based 
on  a process  evaluation  of  program  implementation, 
including  (a)  an  ethnographic  study  of  the  community  which 
included  repeated  key  informant  interviews,  (b)  a 
community  needs/assets  assessment  completed  by  trained 
neighborhood  volunteers,  and  (c)  family  interviews 
conducted  at  the  end  of  the  grant  period.  Finally,  an  impact 
study  which  evaluated  the  relative  costs  of  bringing  services 
into  the  community  and  the  benefit  achieved  for  the 
community  was  conducted. 

The  effectiveness  study  initially  attempted  to  use  a 
quasi-experimental  design  including  a comparison  school 
matched  for  ethnicity  and  percent  free  lunch.  This  was  later 
altered,  dropping  comparison  school  data  and  instead 
attempting  to  develop  matched  comparisons  within  the 
target  schools.  In  addition,  longitudinal  data  was  collected 
on  individual  students.  Nationally  normed  instruments  were 
complemented  by  school  district  data  on  student  attendance, 
discipline  and  achievement.  Changes  in  student  mental 
health,  self-esteem,  and  perceptions  of  school  climate  were 
evaluated.  The  study  sample  included  all  students  enrolled 
in  the  target  schools.  Parental  consent  was  declined  for  a 
small  proportion  of  students  5%). 

Measures  and  Data  CoIIeetion 

The  focus  of  the  School  of  the  Future  is  the  student;  its 
ultimate  goal  is  to  improve  the  quality  of  life  for  children. 
From  earlier  school-based  efforts,  we  know  that  changes  at 
the  school  level  do  not  occur  quickly.  We  looked  for 
changes  at  the  community  level  which  could  be  linked  to  the 
project  as  well  as  changes  in  school  climate,  in  families  who 
received  considerable  direct  services,  and  in  the  physical 
and  mental  health,  self-concept  self  esteem,  and  academic 
achievement  of  students. 

Student  surveys  comprised  the  primary  instrument  for  data 
collection  using  standardized  instruments.  All  students  in 
the  target  schools  served  as  the  sample.  Parent  consent  was 
secured  prior  to  student  inclusion.  Middle  school  students 
annually  completed  a survey  packet  during  a single  class 
period  which  included  Achenbach's  Youth  Self-Report 
(YSR)  version  of  the  Child  Behavior  Checklist  (Achenbach, 
1991a),  the  "School  Life"  section  of  the  National  Education 
Longitudinal  Study  (National  Center  for  Education 
Statistics,  1988),  and  Rosenberg's  Self-Esteem  Measure 
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Statistics,  1988),  and  Rosenberg's  Self-Esteem  Measure 
(Rosenberg,  1965).  School  district  data  was  also  obtained 
including  information  on  grades,  standardized  test  scores, 
and  attendance.  District  data  were  linked  to  individual 
student's  survey  responses.  For  elementary-aged  students, 
teachers  annually  completed  the  Teacher  Report  Form 
version  of  Achenbach's  Child  Behavior  Checklist 
(Achenbach,  1991b).  School  climate  was  measured  using  an 
annual  administration  of  the  "Teacher  School  Climate" 
survey  of  the  National  Education  Longitudinal  Study 
(National  Center  for  Education  Statistics,  1988). 

Results 

Community  Impact 

The  School  of  the  Future  project  was  perceived  as  clearly 
successful  in  all  four  cities.  Though  Hogg  Foundation 
funding  ceased  in  August,  1995,  the  project  has  continued 
and  has  been  replicated  in  each  of  the  original  sites.  More 
specifically  for  Site  A,  the  School  of  the  Future  appears  to 
have  had  considerable  measurable  direct  impact  on  the 
community.  Over  $670,000  in  new  money  for  direct 
services  to  children  and  families  was  generated,  and  more 
than  thirty  new  service  programs  in  areas  of  health,  violence 
prevention,  parenting  support,  mental  health,  recreation,  and 
academic  enrichment  were  secured  and  coordinated.  As 
Figure  1 and  Figure  2 reflect,  about  1 ,000  children  and 
families  per  year  received  direct  services.  When  multiple 
contacts  are  considered,  more  than  1 6,400  instances  of 
service  provision  per  year  were  noted  (65,000  for  the  entire 
project). 

Figure  3 reflects  results  of  a benefit  to  cost  analysis  based 
on  program  component  over  a four  year  period.  Some 
service  components,  such  as  the  health  care  center,  had 
higher  start-up  costs  than  others.  A benefit  to  cost  analysis 
was  also  conducted  for  the  Project  Coordinator  position. 
Though  the  ratio  was  less  than  1 .00  during  the  first  two 
years  of  the  project,  indicating  that  supporting  the 
Coordinator  was  more  expensive  than  the  return  in  dollar 
value  of  services  secured,  by  year  five  that  ratio  exceeded 
20: 1 . For  the  five  year  duration  of  the  project,  the  benefit  to 
cost  ratio  for  the  Project  Coordinator  was  approximately 
4:1 . A number  of  indirect  impacts  were  also  identified 
repeatedly  by  key  informants.  Among  these  were  the 
evolution  of  a neighborhood  collaboration  model  for  the 
city,  development  of  important  linkages  with  other 
organizations,  and  the  development  of  parents  as  leaders  and 
community  advocates.  A conservative  estimate  of  the 
indirect  monetary  value  of  the  School  of  the  Future  project 
for  the  community  was  in  excess  of  $1 .3  million. 

Student  Outcomes 

While  it  can  be  argued  that  the  School  of  the  Future  project 
had  an  important  direct  impact  on  the  community,  the 
question  remained  whether  the  intervention  was  effective  in 
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question  remained  whether  the  intervention  was  effective  in 
terms  of  student  outcomes.  Though  only  one  site  is 
presented  in  this  report,  and  results  are  still  considered 
preliminary,  similar  trends  are  apparent  across  the  other 
sites.  Using  the  norms  associated  with  the  Achenbach  Youth 
Self-Report  (YSR),  students  appeared  much  more  similar  to 
the  referred  sample  than  the  non-referred  during  the  baseline 
period.  This  raised  some  questions  for  the  research  team 
regarding  interpretation.  For  this  reason,  raw  scores  were 
used  rather  than  the  norms  associated  with  the  instrument. 
Table  1 summarizes  results  of  the  YSR.  There  was  no 
sustained  significant  change  as  measured  with  the  YSR  (p  ^ 
.05). 

Student  self-esteem  was  measured  using  the  Rosenberg 
Student  Self  Esteem  Scale.  Two  of  the  ten  items  on  this 
scale  demonstrated  statistically  significant  change  over  the 
three  years  of  administration.  That  change  was  in  the 
desired  direction.  Student  perceptions  of  school  climate 
were  measured  using  the  instrument  from  the  National 
Education  Longitudinal  Study  (NELS).  Eight  of  the  thirteen 
items  in  this  scale  demonstrated  significant  change  over 
time  (p  ^ .05).  Unfortunately,  the  direction  of  change,  in 
each  instance,  was  contrary  to  the  desired  direction. 

Discussion 

The  School  of  the  Future  model  is  essentially  a systemic 
intervention.  There  were  considerable  outcomes  in  the 
community  which  can  be  arguably  linked  to  the 
intervention.  There  were  no  changes  in  student  mental 
health,  self-esteem,  or  school  climate  which  could  be  linked 
to  the  intervention. 

The  use  of  a small  targeted  investment  in  a single  individual 
did  serve  as  a sufficient  catalyst  to  identify,  implement,  and 
coordinate  services;  however,  these  services  have  had  no 
measurable  effect  to  date  on  student  mental  health,  physical 
health,  or  academic  performance.  This  may  be  due  to  such 
factors  as  insufficient  strength  of  the  intervention, 
inadequate  timeline,  inadequate  sensitivity  of  instruments, 
or  inappropriate  sampling  for  evaluation.  There  may  have 
been  a poor  match  between  level  of  intervention  and  design 
of  the  effectiveness  evaluation,  or  this  may  have  been  the 
wrong  intervention  if  change  in  student  mental  health  is  the 
desired  outcome. 
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The  Florida  Inclusion  Network  Survey  Analysis 
and  Implications 
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Introduction 


I The  Regular  Education  Initiative  (REI)  was  first  proposed 

I by  Madeline  Will,  the  former  Assistant  Secretary  for  the 

I Office  of  Special  Education  and  Rehabilitative  Services,  in 

the  mid  1980s.  The  REI  is  a service  delivery  model 
I combining  special  education  expertise  and  general 

I education  expertise.  REI  intended  to  prevent  student  failure, 

i|  instead  of  identifying  failure.  Inclusion  is  a component  of 

I REI.  Inclusion  refers  to  the  successful  participation  of 

I students  having  special  needs  within  the  continuum  of 

regular  education  programs  and  the  community.  Inclusion 
|i  means  modifying  traditional  teaching  methods,  curriculum, 

i and  visions  so  that  a broader  spectrum  of  services  and 

K,  techniques  is  available  to  all  students.  Inclusion  celebrates 

;■  and  accommodates  diversity  by  pooling  resources  and 

p sharing  responsibilities.  By  bringing  together  general  and 

special  education  teachers,  learners  would  receive  the  best 
K of  both  worlds  and  the  number  of  children  with  disabilities 

■:  educated  in  segregated  programs  would  be  greatly  reduced. 


Inclusion  has  taken  on  a variety  of  meanings  and 
perceptions.  Educators  disagree  about  the  nature  of 
inclusion.  Some  proponents  of  inclusion  promote  "full 
inclusion"  for  all  students  and  desire  to  eliminate  the 
continuum  of  services  currently  available.  Others  advocate 
inclusion  as  part  of  a continuum  of  services  designed  for 
individual  students.  Feelings  about  inclusion  vary  greatly 
and  opinions  expressed  about  inclusion  range  from  the 
excitement  of  teaming  to  deliver  services  to  worries  about 
job  loss  and  increased  responsibilities  without  proper 
compensation  or  training.  For  some  educators,  inclusion  is 
the  only  way  to  teach.  For  others,  it  is  one  of  the  ways  to 
serve  learners.  The  only  aspect  of  inclusion  agreed  upon  by 
most  professionals  is  the  need  for  educating  people  about 
inclusion  and  the  need  for  further  research  in  this  area. 
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Method 


The  Florida  Inclusion  Network  sought  to  determine 
professional  knowledge  of  and  feelings  about  inclusion.  A 
total  of  1 80  general  educators,  special  educators, 
administrators,  and  support  personnel  in  the  northeastern 
area  of  Florida  were  surveyed.  These  four  groups  were 
asked  to  respond  to  the  following  questions: 

1 . "What  is  inclusion?" 

2.  "How  do  you  feel  about  inclusion?" 

3.  "How  do  you  think  inclusion  will  affect  you?" 

A total  of  260  surveys  were  sent  out,  with  70%  returned  for 
analysis.  Respondents  were  classified  in  terms  of  the 
position  held  (e.g.,  general  educator),  years  of  experience 
(e.g.,  0-5  years,  6-10  years,  10-20  years,  and  over  20  years), 
and  grade  level  placement  (e.g.,  Pre  K-6th  grade,  middle 
school,  or  high  school).  Answers  for  the  first  question  were 
coded  into  one  of  three  categories:  (a)  knowledgeable,  (b) 
somewhat  knowledgeable,  or  (c)  least  knowledgeable. 
Answers  for  question  two  were  coded  into  one  of  three 
categories:  (a)  feelings  expressed  positively,  (b)  feelings 
expressed  positively  but  with  reservations,  or  (c)  feelings 
expressed  negatively.  Analysis  of  the  answers  to  question 
three  were  categorized  by  content.  Consequently,  survey 
results  were  analyzed  qualitatively  and  quantitatively. 

Due  to  the  categorical  nature  of  the  data,  nonparametric 
measures  were  used  for  analyses.  Alpha  was  preset  at  .01 . 
Researchers  chose  to  be  conservative  due  to  the  large 
sample  size.  A chi  square  procedure,  used  to  detect 
differences  in  terms  of  knowledge  about  inclusion,  showed 
that  not  all  respondent  groups  are  equally  knowledgeable 
about  inclusion,  c2  (6,  N = 1 80)  = 1 9.63 , p < .0 1 . Figure  1 
shows  knowledge  level  across  groups.  Seasoned  educators, 
however,  did  not  differ  from  beginning  teachers  in  terms  of 
their  knowledge  about  inclusion.  Knowledge  about 
inclusion  also  appears  to  be  unaffected  by  grade  level. 

When  analyzing  knowledgeable  responses  of  general  and 
special  educators,  no  apparent  difference  between  the  two 
exists.  When  analyzing  the  least  knowledgeable  responses, 
however,  the  two  groups  do  not  respond  similarly.  When 
comparing  knowledgeable  versus  least  knowledgeable 
responses  for  special  and  general  educators,  a phi  coefficient 
of  .412  demonstrates  a significant  degree  of  association 
between  group  membership  and  level  of  knowledge.  A 
particular  degree  of  knowledge  about  inclusion  is  clearly 
associated  with  whether  respondents  are  general  or  special 
educators. 

An  overall  chi  square  analysis  of  feelings  about  inclusion 
shows  that  groups  differ  on  this  dimension,  c2  (6,  N = 176) 
= 20.7854,  p < .01.  Some  groups  feel  more  strongly  about 
inclusion  than  others,  as  depicted  in  Figure  2.  Again,  years 
of  experience  and  grade  level  placement  do  not  have  a 
significant  impact  on  feelings.  A chi  square,  based  on 
negative  feelings  alone  about  inclusion,  shows  there  are 
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negative  feelings  alone  about  inclusion,  shows  there  are 
differences  among  the  groups,  c2  (3,  N = 64)  = 42.88,  p < 
.01 . The  number  of  negative  responses  is  proportionally 
higher  in  at  least  one  group.  A phi  coefficient  of  .34 
demonstrates  a significant  association  between  group 
membership  and  negative  responses. 

Responses  to  the  third  question,  how  inclusion  affects  the 
respondents,  varied  widely.  The  majority  of  responses  were 
focused  on  pragmatic  issues.  General  educators  were 
concerned  with  limited  planning  time  and  preparation  for 
teaching  students  with  special  needs  within  a general 
classroom  setting.  Comments  include: 

• I feel  I would  be  burdened  more  than  I already  am  in 
planning  and  carrying  out  activities  for  all  children  of 
all  abilities.  I feel  I would  need  to  return  to  school  to 
learn  how  to  deal  with  many  more  child-related 
problems  that  are  prevalent  in  society  today. 

• Inclusion  makes  it  harder  for  me  to  assist  and  help  all 
of  my  students  when  the  disabled/disadvantaged 
student  demands  more  of  me  and  takes  time  away 
from  other  children  who  also  need  me. 

Special  educators  reported  fears  of  job  loss  and  changing 
responsibilities.  Example  responses  include,  "If 
implemented,  I would  become  either  a teacher  on  a rolling 
cart  or  a highly  educated  teacher's  aide,"  and  "Inclusion  will 
eliminate  jobs."  Administrators'  comments  focused  on 
accessing  appropriate  resources  to  implement  inclusion, 
providing  adequate  training  for  teachers,  and  "selling"  the 
idea  to. parents.  On  the  positive  side,  teachers  responded 
favorably  to  a team  approach  to  education,  and  to  the 
opportunity  to  learn  from  each  other.  Example  comments 
from  general  and  special  educators,  respectively,  are: 

• If  there  are  students  having  difficulty  with  a concept 
then  one  of  us  can  take  these  students  and  work  with 
them. 

• Inclusion  would  allow  team  teaching  and  having  an 
opportunity  to  work  in  the  classroom  with  a peer 
would  allow  me  to  learn  more  and  improve 
techniques. 

Implications 

Teacher  education  is  needed  if  inclusion  is  to  remain  a 
system  of  delivery  for  special  needs  students.  Because 
groups  of  educators  differ  in  their  knowledge  about 
inclusion,  instruction  must  be  provided,  so  that  a common 
understanding  of  inclusion  can  be  established. 

Groups  differ  in  terms  of  their  feelings  about  inclusion. 
Negative  feelings,  in  particular,  are  expressed 
disproportionately  across  groups  of  educators.  Reasons  for 
negative  feelings  about  inclusion,  however,  need  to  be 
understood  for  general  and  special  educators  alike.  Do 
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negative  feelings  arise  from  limited  inclusion  knowledge  or 
experience  with  an  unsupported  inclusive  model?  The 
answer  is  unclear.  Further  research  into  inclusion  is 
warranted. 

The  personal  perceptions  of  the  effects  of  inclusion  vary. 
Concerns  expressed  by  educators  reveal  the  need  for 
implementing  appropriate  models  of  inclusion,  if  inclusion 
is  to  survive  into  the  twenty-first  century.  The  concerns 
expressed  by  sampled  educators,  such  as  impact  on  planning 
time,  job  security,  and  changing  responsibilities,  must  not 
be  overlooked.  Inclusion  is  not  the  only  answer  to  problems 
encountered  within  a dual  system  of  education.  It  is, 
however,  a delivery  system  that  must  be  carefully 
considered  in  the  education  of  students  with  disabilities. 
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Staff  Development  and  the  Inclusion  of  Middle 
School  Students  with  EBD  in  Regular 
Education:  Preliminary  Findings  from  Project 
DESTINY 

Introduction 


Including  students  with  Emotional  or  Behavioral  Disorders 
(EBD)  in  general  education  classrooms,  while  desirable  in 
many  respects,  has  been  controversial  among  those 
concerned  with  education  and  children's  mental  health 
(Fuchs  & Fuchs,  1994;  MacMillan,  Gresham,  & Fomess, 
jr  c 1996;  U.S.  Department  of  Education,  1995).  Critics  have 

^ .ill::  .Ur.  ■ suggested  that  this  practice  may  not  be  in  the  best  interests 

of  the  students,  their  classmates,  or  the  school  community  as 
a whole  (Kauffman,  1993).  A specific  concern  has  been 
that,  in  order  to  effectively  educate  students  with  EBD, 
general  educators  need  to  broaden  their  knowledge  in  this 
area  (Jones  & Jones,  1990;  Knitzer,  1982;  Knitzer, 

Steinberg,  & Fleisch,  1990). 

li.  :!  . Enhancing  general  educators'  knowledge  in  the  area  of  EBD 

is  one  of  several  issues  being  investigated  in  Project 
DESTINY  (Designing  Educational  Support  Teams  through 
Interdisciplinary  Networks  for  Youth  with  Emotional  or 
Behavioral  Disorders),  a three  year  research  project,  funded 
by  the  U.S.  Office  of  Special  Education  and  Rehabilitative 
Services  (Barringer  & Cheney,  1995;  Cheney  & Barringer, 
in  press)  and  designed  to  evaluate  models  for  school-based 
service  delivery  to  students  with  EBD.  This  summary 
concerns  our  preliminary  findings  regarding:  (a)  the 
knowledge  and  skills  of  teachers  from  two  of  our 
intervention  schools;  (b)  the  effects  of  two  types  of  staff 
development  activities  (i.e.,  didactic  presentations  and  case 
studies  on  teachers'  perceptions  of  their  competence  in  the 
area  of  EBD);  and  (c)  the  effects  of  increases  in  teachers' 
perceptions  of  their  own  competence  on  the  academic  and 
social  functioning  of  their  students. 
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Method 


Setting 

Data  were  collected  at  two  New  Hampshire  middle  schools. 
School  A,  with  an  enrollment  of  520  students  (grades  6-8), 
is  located  in  a community  of  9,500  residents,  98%  of  whom 
are  Caucasian.  School  B is  located  in  a community  of 
75,000,  also  primarily  Caucasian  (96%),  enrolling  875 
students  (grades  7-9). 

Participants 

Participants  from  School  A included  27  general  education 
teachers  and  3 special  educators,  with  an  average  of  1 7 
years  of  experience.  A total  of  2 1 had  Masters  degrees  and  9 
had  Bachelor's  degrees.  In  school  B,  there  were  30  general 
and  4 special  education  teachers,  with  an  average  of  1 5 
years  of  experience.  A total  of  20  had  Masters  degrees,  and 
14  had  Bachelors  degrees. 

Using  the  Systematic  Screening  for  Behavior  Disorders 
(SSBD;  Walker  & Severson,  1992),  teachers  identified 
students  as  either  typically  developing  or  having  EBD. 
Requests  for  permission  to  participate  in  the  project  were 
mailed  to  parents  of  the  children  whose  scores  were  the  top 
three  on  internalizing  and  externalizing  scales  from  each 
class  as  well  as  parents  of  classmates  considered  typical  by 
their  teachers.  Students  having  permission  were  then 
assessed  using  the  rating  scales.  Finally,  groups  of  students 
were  designated  based  on  the  results. 

Teachers,  students,  and  parents  then  completed  the 
appropriate  Achenbach  (1991)  behavior  checklist  and  the 
Social  Skills  Rating  System  (SSRS;  Gresham  and  Elliott, 

1 990).  Students  also  completed  the  Reynolds  Adolescent 
Depression  Scale  (RADS;  Reynolds,  1989).  Students  whose 
T-scores  were  greater  than  60  on  any  of  the  Achenbach 
broad  band  scales,  or  above  70  on  the  RADS,  were  placed  in 
the  group  with  EBD.  A total  of  22  students  with  EBD  (14 
males,  8 females;  age  x = 12.9)  were  identified  in  school  A, 
and  20  students  with  EBD  (12  males,  8 females;  age  x = 
13.2)  were  identified  in  school  B.  Twenty-seven  students 
were  identified  as  typical  for  their  age  and  grade  in  school  A 
(14  males,  13  females;  age  x = 12.6),  and  19  were  identified 
in  School  B (8  males,  1 1 females;  age  x = 13.3);  all  of  these 
students  scored  within  the  average  range  on  the  screening 
measures. 

Procedure 

During  the  1994-95  school  year,  the  Project  DESTINY 
Team  (i.e.,  a clinical  psychologist,  a special  educator,  and  a 
family  social  worker)  engaged  in  two  types  of  staff 
development:  (a)  the  first  being  a series  of  eight  didactic 
workshops,  on  such  topics  as  developing  partnerships  with 
families,  clinical  syndromes,  and  crisis  intervention 
techniques  (Cheney  & Barringer,  in  press);  and  (b)  a series 
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techniques  (Cheney  & Barringer,  in  press);  and  (b)  a series 
of  bi-monthly  case  studies  of  students,  which  included 
members  of  the  school  staff,  and,  whenever  possible,  the 
parents  and  students  themselves. 

Both  schools  participated  in  the  didactic  training  and  were 
scheduled  to  complete  case  studies.  However,  due  to 
leadership  issues  and  contract  negotiations,  only  School  A 
participated  in  case  studies.  The  case  studies  were  highly 
structured  meetings  intended  to  develop  individualized 
plans  for  students  with  EBD.  Information  concerning  4 
developmental  factors  (biological,  interpersonal,  affective, 
and  cognitive)  was  collected  using  teacher,  student,  and 
parent  versions  of  the  same  data  collection  forms  (a  sample 
of  the  student  form  is  presented  in  Figure  1).  Using  this 
information,  staff,  students,  and  parents  then  met  to  identify 
those  areas  of  the  student's  development  in  which  support 
was  needed-.  Student  supports  were  specified  in  each 
relevant  developmental  area,  and  dates  were  set  for  review 
in  order  to  assess  the  effectiveness  of  these  supports.  In 
contrast,  teachers  in  school  B met  informally  with  the 
consultants,  raising  questions  about  students  of  concern. 
Students  and  parents  did  not  participate  in  these  meetings. 

Measures 

In  both  schools,  the  measures  discussed  below  were 
administered  at  the  beginning  and  end  of  the  school  year. 

Teacher  Knowledge  and  Skills  Survey  (TKSS).  Based  on 
two  recently  published  lists  of  competencies  for  educators 
of  students  with  EBD  (Braaten,  1993;  Bullock,  Ellis,  & 
Wilson,  1994),  a 45  item  measure  of  teachers'  perceptions  of 
their  knowledge  and  skills  related  to  EBD  was  constructed. 

A factor  analysis  of  these  items  from  1 14  teacher  surveys 
identified  six  factors,  which  included  36  items  that 
accounted  for  74%  of  the  variance.  The  six  factors  were 
used  as  subscales,  all  of  which  had  both  acceptable  internal 
consistency  (five  scales  exceeded  .8  and  one  scale  alpha  = 
.75)  and  item  to  total  correlations  (.4  - .7). 

Students'  social  and  emotional  functioning  were  measured 
using  two  rating  scales:  the  Teacher  Rating  Form 
(Achenbach,  1991),  a 1 13  item  scale,  that  assesses 
internalizing  and  externalizing  behavior  problems,  and  the 
Social  Skills  subscale  of  the  Social  Skills  Rating  System 
(Gresham  & Elliott,  1990),  which  measures  student  progress 
on  30  desirable  behaviors.  In  addition,  students'  grade  point 
averages  were  computed  for  core  classes  in  Math,  Language 
Arts,  Social  Studies,  and  English,  and  attendance  was 
recorded  as  days  absent  across  a 175  day  school  year. 

Results 

Teacher  Knowledge  and  Skills 

Pre-  and  post-test  subscale  scores  for  teachers  from  schools 
A and  B are  included  in  Table  1.  A MANOVA  indicated 
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A and  B are  included  in  Table  1.  A MANOVA  indicated 
significant  pre-test  differences  between  the  two  groups  of 
teachers  (Hotellings  T2  = .77,  df  = 6,58,  p < .01)  on  all  but 
one  factor  (i.e..  Classroom  Behavioral  Assessment).  In 
general,  teachers  from  School  A rated  themselves  as  more 
knowledgeable  than  those  from  School  B on  the  pre-test 
administration  of  the  TKSS.  With  the  exception  of  the 
Classroom  Instruction  subscale  of  the  TKSS,  teachers  from 
School  B had  a mean  rating  below  3 (moderate  level  of 
competence)  on  a five  point  scale  for  all  knowledge/skill 
areas.  The  mean  score  for  teachers  from  School  A was 
below  3 on  the  Theory  and  Characteristics,  Background  and 
Eligibility,  and  Classroom  Behavioral  Assessment 
subscales. 

A subsequent  MANCOVA  with  repeated  measures,  using 
teachers'  pre-test  scores  as  covariates,  indicated  significantly 
higher  post-test  scores  (Hotellings  T2  = .35,  df  = p < .01) 
for  teachers  from  school  A.  Univariate  tests  showed  that 
these  differences  were  significant  (p  < .05)  for  all  subscales 
of  the  TKSS  except  Classroom  Behavioral  Assessment. 

Teachers'  Ratings  of  Students 

In  order  to  discern  the  impact  of  teacher  education  on 
student  performance,  student  measures  were  examined. A 
MANOVA  with  repeated  measures,  using  TRF  and  SSRS 
data,  also  indicated  significant  pre-test  differences  between 
schools  on  the  Social  Skills  subscale  (F  = 16.08,  df  = 1,76,  p 
< .001),  as  well  as  a significant  Student  Group  x School 
interaction 

(F  = 9.82,  df  = 1,  76,  p < .01).  In  both  schools,  typically 
functioning  students  scored  higher  than  students  with  EBD 
on  measures  of  social  skills,  but  this  difference  between 
student  groups  was  significantly  greater  in  School  B than  in  i 
School  A.  The  results  of  a MANCOVA,  covarying  pre-test 
differences  between  schools,  showed  only  the  expected 
significant  post-test  difference  between  students  identified 
as  having  EBD  and  those  identified  as  typically  functioning. 

Attendance  and  Grades 

Students'  attendance  and  grades  were  also  examined  as 
indicators  of  the  effect  of  teacher  education  on  student 
performance.  In  School  A,  the  attendance  of  students  with 
EBD  (M  = 172.56,  SD  = 5.3)  was  comparable  to  the 
attendance  of  students  in  the  typical  group  (M  = 1 71 .73,  SD 
= 6.9).  In  contrast,  students  with  EBD  from  School  B 
attended  school  fewer  days  (M  = 164.88,  SD  = 13.90)  than 
students  in  the  typical  group  (M  = 173.70,  SD  = 5.6).  An 
ANOVA  of  the  absence  data  yielded  a significant  Group  x 
School  interaction  (F=  4.57,  df  = 1,78,  p < .05). 

As  expected,  the  grade  point  averages  of  students  with  EBD 
were  generally  lower  than  those  of  students  in  the  typical 
group  across  schools  (see  Table  2).  Mean  grade  point 
averages  across  quarterly  marking  periods,  however,  were 
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averages  across  quarterly  marking  periods,  however,  were 
stable  for  both  groups  of  students  in  school  A,  but  only  for 
the  group  of  typically  functioning  students  in  School  B.  In 
School  B,  students  with  EBD  showed  both  considerable 
variability  in  their  mean  grade  point  averages  across  the  four 
marking  periods,  and  a significant  decline  (F  = 4.37,  df  = 1, 
76)  from  the  first  to  the  fourth  quarter.  Using  first  quarter 
grades  in  a subsequent  MANCOVA  with  repeated 
measures,  however,  failed  to  yield  significant  differences. 

Discussion 

Despite  the  obvious  design  limitations,  preliminary  analyses 
of  these  data  have  been  useful  in  pointing  out  the  nature  of 
differences  between  the  participants,  and  in  shaping  our 
staff  development  activities  accordingly.  With  regard  to  the 
latter,  our  preliminary  findings  suggest  that  not  all  staff 
development  methods  provide  the  same  results.  When 
combined  with  traditional  in-service  presentations,  the  case 
study  component  of  the  model  seemed  to  enhance  outcomes 
for  staff  members.  Thus,  our  greatest  success  was  in  school 
A,  in  which  the  leadership  demonstrated  a strong 
commitment  to  regular  meetings  involving  teams  of 
teachers,  students,  and  parents.  These  meetings  were 
devoted  to  sharing  information,  developing  individualized 
action  plans  within  a structured,  developmental  framework, 
and,  in  so  doing,  building  important  relationships  among  the 
participants.  When  compared  to  students  in  school  B, 
students  with  EBD  in  school  A had  higher  grades,  less 
fluctuation  in  grades  across  marking  periods,  and  higher 
attendance  rates. 

The  case  study  process  was  also  designed  to  increase  the 
accuracy  of  teachers'  attributions  regarding  the  causes  of 
behavioral  problems.  We  know  that  teachers  who  attribute  a 
student's  behavior  to  factors  beyond  the  student's  control 
(e.g.,  family  relationships,  bio-medical  issues,  financial 
hardship,  etc.),  are  generally  more  supportive  than  those 
who  regard  all  rule  violations  as  strictly  volitional  (Brophy, 
1985).  With  respect  to  this  issue,  teachers  from  School  A 
reported  that  case  studies  left  them  with  both  a deeper 
understanding  of  the  material  presented  in  didactic 
workshops  and  a more  differentiated  picture  of  students  and 
their  families. 

Based  on  their  deeper  understanding  of  the  students'  and 
families'  unique  needs,  teachers  appeared  better  able  to 
selectively  and  appropriately  use  the  tools  that  had  been 
presented  in  didactic  seminars.  Our  preliminary  findings 
suggest  that  the  plans  developed  during  case  studies  had 
little  impact  on  teachers'  evaluations  of  students' 
social-emotional  functioning,  but  may  have  influenced  both 
attendance  and  grades.  During  the  second  year  of  Project 
Destiny,  we  have  been  collecting  data  that  bear  more 
directly  on  the  relationship  between  our  measure  of  staff 
development  and  measures  of  students'  functioning.  In 
addition,  we  have  added  control  schools  to  this  year's 
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design. 
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Introduction 

The  school  system  has  been  underutilized  as  an  avenue  to 
obtain  the  objectives  of  the  Healthy  Children  2000 
document  which  contains  national  objectives  and  strategies 
for  significantly  improving  the  health  of  children  (Public 
Health  Service,  1991).  In  particular,  the  provision  of  mental 
health  services  within  the  context  of  the  school  environment 
enhances  the  accessibility  and  normalization  of  mental 
health  services  to  populations  of  children  and  their  families 
(Mash  & Barkley,  1989;  Meeker,  DeAngelis,  Berman, 
Freedman  & Oda,  1986;  Office  of  Disease  Prevention  and 
Health  Promotion,  1993).  The  concept  of  health  would  be 
inclusive  of  psychosocial  and  emotional  needs  without 
stigmatization. 

The  purpose  of  this  study  was  to  develop,  test,  and  evaluate 
a mental  health  intervention  program  implemented  within 
the  school  environment.  The  particular  program  entitled 
"Self-Management  Intervention  Program  for  School-Age 
Children  with  Chronic  Health  Conditions  and  Their 
Parents"  focused  on  the  promotion  of  self-regulatory  and 
stress  management  skills  to  enhance  coping  with  a chronic 
health  condition.  The  program  was  carried  out  immediately 
after  school  hours  and  involved  3 components:  (a)  twelve 
child  group  sessions;  (b)  three  parent  group  sessions;  and  (c) 
1-2  home  visits. 
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Method 


A pre-  and  post-test  design  was  used  to  examine  child, 
family,  and  system  outcomes.  Program  outcomes  were 
collected  at  various  time  points  to  represent  short  term  and 
long  term  effects.  The  target  group  included  children  7 to  14 
years  of  age  with  a chronic  health  condition  of  six  months 
or  longer  duration  and  identified  by  the  school  nurse  as 
having  difficulty  coping  with  the  stresses  associated  with 
the  health  condition.  A total  of  sixty-five  children 
participated  in  the  program,  representing  17  different 
schools  in  two  school  districts. 

Six  program  implementation  and  evaluation  issues  with 
associated  resolutions  will  be  discussed. 

Implementation  and  Evaluation  Issues 

The  first  issue  relates  to  the  non-categorical  approach  to 
programs  that  are  disseminated  in  the  community  such  as 
the  school  system.  Typically,  within  the  health  care  system, 
programs  are  organized  around  diagnostic  groupings  related 
to  physical  and  mental  health  disorders.  However,  when 
implementing  programs  within  the  community,  it  is  often 
not  feasible  to  use  categorical  groupings  due  to  the  few 
children  found  in  any  one  category.  From  a theoretical 
perspective,  "generic  functional  skills"  such  as  stress 
management  and  coping  are  relevant  for  many  disorders; 
thus,  it  may  be  more  important  to  organize  programs  around 
functional  issues  rather  than  the  disorder.  Stigmatization  is 
also  minimized  when  programs  emphasize  life  skills.  In  this 
self-management  program,  children  had  a variety  of  chronic 
physical  health  conditions  associated  with  mental  health 
symptoms.  The  program's  intervention  focused  on  the 
self-regulatory  skills  of  self-observation,  self-monitoring, 
self-instruction,  self-reinforcement,  and  self-evaluation.  The 
intervention  also  addressed  emotion-focused  and  problem- 
focused  stress  management  skills.  When  using  a 
non-categorical  approach,  several  program  design  and 
evaluation  challenges  need  to  be  considered.  Variability 
within  treatment  groups,  applicability  to  one's  specific 
condition,  and  generalizability  to  the  family  home 
environment  pose  several  challenges.  Specific  program 
components  were  incorporated  in  this  study  to  address  these 
challenges.  For  example,  a family  home  visit  was  made  to 
each  family  to  develop  individualized  goals  associated  with 
the  school  based  program  goals.  During  each  group  session, 
attention  was  paid  to  individualized  goals. 

The  second  issue  relates  to  accessibility  and  normalization. 
The  program  was  implemented  within  a school  building 
located  within  the  child's  neighborhood  and  was 
supplemented  by  1-2  home  visits.  Interviews  with  the 
children  and  parents  consistently  confirmed  that  the 
convenient  location  of  the  program  and  lack  of  stigma 
associated  with  the  school  program  setting  contributed  to 
success  in  recruitment  and  the  nearly  100%  attendance  rate. 
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success  in  recruitment  and  the  nearly  100%  attendance  rate. 
Children  were  more  responsive  to  teachers'  and  counselors' 
encouragement  to  enroll  in  the  program  because  its  setting 
was  school  based,  accessible,  and  did  not  have  the  stigma 
often  associated  with  a "mental  health  center." 

The  third  issue  focuses  on  program  integrity  for  both 
process  and  content.  Program  integrity  is  particularly 
important  because  multiple  groups  were  conducted  in 
different  school  sites.  Assuring  program  integrity  is  a 
challenge  when  the  intervention  program  is  carried  out  in  a 
school/community  based  environment.  A curricular  manual 
was  developed  with  several  measures  of  integrity  obtained 
for  group  process  variables  (cohesion,  cooperation,  support, 
etc.),  educational  process  components  (instruction, 
demonstration,  application  to  home  environment, 
reinforcement,  etc.),  and  program  content  (specific 
self-regulatory  and  stress  management  knowledge  and 
skills).  By  measuring  program  integrity,  the  variability  in 
the  program  process  and  content  was  monitored  and 
analyzed  by  various  statistical  analyses.  The  relationship 
between  program  outcome  results  and  program  integrity 
could  then  be  determined. 

The  fourth  issue  involves  the  conceptualization  of  program 
outcomes.  The  selection  of  program  outcomes  should 
capture  the  comprehensive  nature  and  inter-relationship 
among  the  complexity  of  effects  relevant  to  the  child, 
family,  and  system.  Program  outcomes  were  categorized  as: 
(a)  child  focused  (behavioral-emotional  problems,  self 
system,  health  behaviors,  and  symptoms  of  stress);  (b) 
family  focused  (relational  and  functional);  and  (c)  system 
focused  (school  attendance,  number  of  health  visits  to 
school  nurse  and/or  primary  care  physician).  Program 
outcomes  were  conceptualized  as  primary  and  secondary  in 
that  secondary  outcomes  would  occur  if  positive  change  had 
occurred  in  the  primary  outcomes.  For  example,  a positive 
change  on  a child's  self-regulatory  knowledge  and  skills 
coupled  with  fewer  symptoms  of  stress  could  be  associated 
with  the  secondary  effects  of  fewer  visits  to  the  school 
nurse.  Both  short-term  and  long-term  effects  were  examined 
for  patterns  across  time  and  for  the  sequencing  of  primary 
versus  secondary  outcomes. 

The  fifth  issue  relates  to  the  generalization  and  transfer  of 
program  effects  to  daily  living  at  home  and  in  school.  Two 
inter-related  questions  were  asked.  Did  children  demonstrate 
new  knowledge  and  skills  within  the  program  sessions?  If 
so,  to  what  extent  did  the  children  apply  their  newly 
obtained  knowledge  and  skills  to  daily  living  at  home  and 
school?  In  other  words,  was  the  child  able  to  transfer 
learning  from  the  program  sessions  to  other  settings?  These 
evaluation  questions  were  answered  by  obtaining  data  from 
parents,  teachers,  and  school  nurses.  The  schools  who 
participated  in  the  study  varied  in  the  type  of  community 
based  environments  offered  to  children  including  mental 
health  education  and  clinical  services.  Thus,  the  availability 
of  school  resources  designated  to  support  the  study's 
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intervention  program  influenced  the  extent  that  the 
intervention  generalized  to  the  school  environment.  Parents 
indicated  that  periodic  booster  sessions  after  completion  of 
the  intervention  would  have  been  helpful  in  order  to  support 
and  sustain  the  newly  learned  behavioral  patterns  over  time. 

The  last  issue  relates  to  individual  differences,  differential 
program  effects,  and  program  matching.  Designing 
programs  both  for  populations  and  for  subgroups  within 
populations  may  be  a way  to  resolve  the  delicate  balance 
issue  of  recognizing  individual  variations  within 
populations  and  thus  matching  variations  within  program  to 
variations  within  populations.  The  variations  in  the  schools' 
emphasis  on  mental  health  resources  and  services  reflected 
to  some  extent  the  community's  attitudinal  and  financial 
support  of  these  services.  The  design  and  efficacy  of  school 
based  intervention  programs  need  to  take  into  consideration 
the  broader  context  of  the  community. 

As  indicated  in  the  federal  document.  School  Health: 
Findings  from  Evaluated  Programs  (1993),  school  health 
programs  and  evaluations  varied  considerable  across  the 
nation.  While  ideally  school  health  programs  need  to 
address  the  full  range  of  health  promotion  services, 
including  mental  health,  the  majority  of  school  health 
programs  are  not  comprehensive.  Program  design  and 
evaluation  need  to  be  considered  carefully  in  order  to 
generate  a data  base  that  offers  insight  into  how  school 
programs  may  assist  families  and  communities  in  meeting 
the  many  needs  of  school-aged  children. 
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The  Elementary  Mental  Health  Operations 

Evaluation 

jbaJ 

1 

Introduction 

A 1986  Needs  Assessment  report  conducted  for  the  Juvenile 
Welfare  Board  (JWB),  an  independent  special  taxing  district 
that  funds  delivery  of  children's  services  through  other 
agencies,  identified  Pinellas  County  Florida  as  an  area 
where  children  with  emotional  handicaps  (EH)  were 
undermet.  In  response,  JWB  began  funding  four  Elementary 
Mental  Health  Operations  (ELMHO)  programs  in  Pinellas 
County.  The  purpose  of  the  ELMHO  programs  was  to 
provide  school-based  mental  health  services  to  the  children 
and  their  families  to  help  elementary  students  with 
emotional  disabilities  learn  to  cope  with  their  issues  well 
enough  to  mainstream. back  into  less  restrictive  academic 
settings.  The  program's  intentions  were  to  demonstrate  the 
effectiveness  of  intensive  mental  health  intervention  with 
this  EH  population  with  "mild  to  moderate"  disabilities 
through  improvements  in  behavior,  increased 
mainstreaming,  and  decreased  placement  in  more  restrictive 
settings. 
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Method 


Subjects/Sites 

There  were  four  program  sites;  three  of  the  sites  involved 
students  with  emotional  disabilities  while  the  fourth  site  was 
moved  within  months  of  inception  to  a center  for  students 
with  severe  emotional  disturbances  (SED)  as  a result  of  the 
school  system  relocating  the  students. 

Intervention/Program 

The  programs  would  address:  (a)  the  child's  behavioral 
problems  and  level  of  functioning  through  individual, 
group,  and  family  counseling;  (b)  identified  family 
problems  through  parent  training  and  family  counseling; 
and  (c)  consultation  with  teachers  to  help  them  develop 
problem  solving  strategies  to  work  with  these  children. 

The  ELMHO  program  process  was  simple  and 
straightforward.  After  a teacher  referred  the  child,  the 
therapist  assessed  the  eligibility  of  that  child.  If  parental 
permission  was  received,  individual,  group,  and  family 
counseling,  case  management,  and 

psychiatric/psychological  examinations  were  provided.  The 
child  could  exit  the  program  for  reasons  such  as 
mainstreaming,  "graduating"  to  middle  school,  leaving  the 
school,  or  lack  of  progress.  The  program  provided  services 
in  a natural  setting,  thought  to  be  less  threatening  to  families 
than  agency-based  mental  health  services. 

Measurement 

The  evaluation  of  this  program  consisted  of:  (a)  interviews 
including  those  with  parents,  school  principals,  social 
workers,  program  supervisors,  agency  directors,  and  JWB 
staff  (N  = 62);  (b)  school  record  reviews  of  all  EH  students 
at  3 target  schools,  a sample  of  SED  students  at  the  SED 
center,  and  all  EH  students  at  two  non-ELMHO  schools  (N 
= 396);  (c)  case  record  reviews  (N  = 100);  and  (d)  case 
studies  ^ = 11). 

Results 

Length  of  Stay 

Participants  stayed  in  the  program  longer  than  was 
originally  intended.  Evaluation  data  suggest  that  children 
with  the  most  severe  rather  than  mild  to  moderate 
disabilities  were  referred  to  the  program.  These  children 
needed  long  term  care  beyond  the  prevention  and  early 
intervention  components  the  ELMHO  program  was 
intended  to  provide.  This  excessive  length  of  stay  also 
reflected  an  apparent  dependency  on  the  program.  Many 
ELMHO  children  reported  not  being  ready  to  leave  EH 
classrooms,  and  some  reported  that  they  intentionally 
misbehaved  to  stay.  Parents  expressed  concern  over  what 
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misbehaved  to  stay.  Parents  expressed  concern  over  what 
they  would  do  when  their  children  were  promoted  to  middle 
school. 

Parental  Participation 

The  evaluation  showed  that  involvement  of  the  whole 
family  in  the  interventions  was  critical  to  significant 
progress.  If  parents  did  not  "buy  into  the  program,"  then  the 
child  could  be  confused  by  the  expressed  wishes  of  the 
parents/family  and  the  wishes  of  the  program,  limiting  or 
preventing  bonding  among  participants.  It  appeared  that 
many  of  the  families  being  served  by  the  programs  had  a 
multitude  of  issues,  making  work  with  families  difficult 
outside  of  the  structured  classroom  setting. 

Findings  showed  that  more  families/siblings  needed  to  be 
involved  in  these  programs.  Service  level  numbers  and 
comments  by  families  indicated  that  children  were  often 
being  served  in  isolation.  Given  the  issues  surrounding  these 
families,  nontraditional  parent  involvement  strategies  may 
need  to  be  employed.  It  appears  that  the  programs  should 
also  support  development  of  community  support  services 
outside  of  mental  health  agencies  so  that  families  are  able  to 
independently  seek  additional  help  in  the  community. 

Service  Need 

Accessing  families  through  the  school  is  a non-threatening 
way  to  reach  individuals  who  might  otherwise  not  seek  help. 
There  appeared  to  be  a need  for  an  increase  in  both  the 
amount  and  types  of  mental  health  services  available  in  this 
setting  for  children  identified  with  an  EH  educational  status. 
The  level  of  services  provided  by  the  schools  did  not  always 
meet  the  identified  need  of  the  child,  because  resources 
remain  scarce  within  the  school  system. 

School  System  Issues 

According  to  the  school  system,  there  was  no  clear  system 
in  place  to  track  students  through  the  mainstreaming 
process.  This  made  the  development  of  some  goals  and 
objectives  for  ancillary  programs,  such  as  ELMHO,  more 
difficult.  As  it  currently  operates,  the  goal  of  mainstreaming 
may  not  be  a measurable,  realistic  objective  for  this 
program.  Methods  to  recruit  and  serve  children  with  less 
severe  disabilities  need  to  be  developed,  the  desired 
outcomes  need  to  be  redefined,  and/or  a new  approach  to 
serving  the  intended  target  families  needs  to  be  considered. 

Another  issue  is  the  need  for  trained  teachers.  According  to 
the  school  district  informants,  there  is  a critical  shortage  of 
exceptional  education  teachers.  Children  can  be  negatively 
impacted  by  limited  teacher  training. 

A final  issue  is  that  of  appropriately  ending  services  to 
children  and  families.  The  ELMHO  program  served  the 
majority  of  the  children  with  emotional  handicaps  in  these 
four  schools,  making  it  difficult  to  close  cases  on  children 
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four  schools,  making  it  difficult  to  close  cases  on  children 
who  have  not  left  the  classroom  and/or  school.  The  children 
may  not  understand  why  they  no  longer  receive  "special 
attention." 

Service  System  Issues 

Concern  arises  in  regard  to  labeling  young  children  with 
both  mental  health  diagnoses  and  exceptional  identification 
by  the  schools.  Labeling  can  lead  to  self-fulfilling 
prophecies  for  failure  in  the  client.  By  focusing  on  children 
with  less  severe  levels  of  disability,  labeling  may  be 
prevented. 

Mental  health  agencies  have  begun  to  bill  Medicaid, 
insurance,  and/or  families  for  the  ELMHO  services.  Several 
concerns  regarding  this  practice  include:  (a)  the  program 
was  originally  free  to  families  and  is  still  seen  as  such;  (b) 
there  is  a need  for  a diagnosis/label  to  obtain 
reimbursement;  and  (c)  the  potential  exists  to  have  clients 
selected  on  the  basis  of  income  source,  or  to  have  treatment 
plans  designed  around  what  Medicaid/insurance  will  pay 
rather  than  on  the  needs  of  the  client. 

Carryover  of  clients  from  one  year  to  the  next  was  also 
found  to  be  excessive  and  distorted  true  service  levels  due  to 
a lag  in  record  keeping.  Cases  closed  during  this  program 
year  at  one  site  averaged  3 1 months  from  the  last  contact 
and  the  case  being  closed.  Cases  at  other  sites  ranged  from  5 
months  to  12  months  between  last  contact  and  cases  closing. 
Some  of  this  lag  was  due  to  delaying  paperwork  until  the 
slower  summer  months  of  the  program. 

The  SED  Center 

The  SED  Center  did  not  appear  to  be  an  appropriate  site  for 
the  ELMHO  program,  for  a number  of  reasons: 

• Documentation  (i.e.,  medical  history,  lost  records, 
etc.)  required  by  the  program  was  consistently 
deficient. 

• These  children  had  more  severe  impairments  than  the 
population  that  the  program  model  was  designed  to 
serve.  The  center  had  a very  structured  environment; 
the  center's  point  and  level  systems  did  not  provide 
teachers  the  flexibility  to  adequately  accommodate 
individualized  treatment  goals.  Children  were 
required  to  complete  two  transitions  before  going 
back  to  their  regular  school,  making  this  process  long 
and  difficult  for  these  children. 

• Most  of  the  children  in  the  SED  program  were 
Medicaid  eligible.  As  before  mentioned,  there  are 
concerns  surrounding  utilization  of  Medicaid-driven 
rather  than  client-driven  admission  and  service  levels. 

Discussion 
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The  original  intent  of  the  ELMHO  program  was  to  provide 
prevention  and  early  intervention  for  children  newly 
diagnosed  as  having  emotional  disabilities  to  help  stabilize 
and  mainstream  them  back  into  regular  education  classes.  It 
slowly  evolved  into  a program  which  appeared  to  serve 
children  with  the  most  severe  problems,  providing 
stabilization  for  them  until  they  "graduate"  to  middle 
school.  After  leaving  the  program,  the  gains  made  while  in 
the  program  appeared  to  be  lost,  and  the  children's  problems 
continued  to  increase  in  severity.  The  "drift"  between 
program  intent  and  function  clearly  needs  to  be  addressed. 

A beginning  point  might  include  the  development  of  a 
problem  solving  team,  including  both  representatives  from 
JWB  and  the  mental  health  agencies,  with  input  from  the 
school  system. 
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Chapter  10 


Assessment  of  Service  & Training  Needs 


407 


The  9th  Annual  Research  Conference  Proceedings,  A System  of  Care  for  Children's  Mental  Health:  Expanding  the 

Research  Base  (February  26  - 28,  1996). 


Education  and  Training  of  Children's  Mental 
Health  Professionals:  The  Existing  and 
Potential  Role  of  Schools  of  Public  Health 
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Introduction 

In  the  early  1980s,  a landmark  national  study  by  Knitzer 
(1982),  conducted  on  behalf  of  the  Children's  Defense  Fund, 
found  a widespread  failure  of  states  to  meet  the  service 
delivery  needs  of  children  and  adolescents  with  serious 
emotional  disorders.  In  that  study,  only  2 1 states  were  found 
to  have  a child  and  adolescent  mental  health  unit,  and  it  was 
estimated  that  approximately  two-thirds  of  children  with 
serious  emotional  illnesses  were  not  receiving  needed 
services. 

Since  the  publication  of  this  study,  increasing  attention  has 
been  given  in  the  literature  to  the  conceptualization, 
development,  and  implementation  of  integrated  and 
comprehensive  mental  health  systems  of  care  for  children 
and  adolescents  (Bums  and  Friedman,  1990;  Friedman  and 
Kutash,  1992;  Jordan  and  Hernandez,  1990;  and  Stroul  and 
Friedman,  1986).  Davis,  Yelton,  Katz-Leavy,  and  Lourie 
(1995)  recently  reported  significant  progress  in  the 
organization  and  support  of  children  and  adolescent  mental 
health  services  due,  in  part,  to  initiatives  by  private 
foundations,  the  passage  of  federal  legislation,  increased 
interagency  collaboration,  state  legal  mandates  and  court 
orders,  and  expansion  in  the  availability  of  alternative 
mental  health  services. 

Meanwhile,  within  the  educational  systems  throughout  the 
United  States,  research  has  suggested  that  educational 
services  for  children  with  emotional  and  behavioral 
disabilities  have  been  inadequate  and  have  resulted  in  poor 
outcomes  for  children  (Knitzer,  1982;  Knitzer,  Steinberg,  & 
Fleisch,  1990;  Knitzer,  1993;  and  Wagner,  D'Amico, 
Marder,  Newman,  & Blackorby,  1991).  According  to  the 
Office  of  Special  Education  and  Rehabilitative  Services 
(U.S.  Department  of  Education,  1992),  these  children  have 
lower  graduation  rates,  lower  grade  point  averages,  and  are 
less  likely  to  continue  in  school  compared  to  all  students. 
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Prompted  by  these  findings  and  in  response  to  the  need  for 
improving  the  outcomes  for  children/adolescents  and  their 
families,  a shift  in  both  the  conceptualization  and  practice 
within  an  integrated  mental  health  system  of  care  is  in 
progress.  The  recent  national  study  reported  by  Davis  et  al. 
(1995)  found  significant  progress  nationally  (since  1982)  in 
the  organization,  financing,  and  delivery  of  mental  health 
services  to  children  and  adolescents.  Nevertheless,  in 
considering  this  new  children's  mental  health  paradigm  of 
community-based  systems  of  care,  Knitzer  (1993)  and  Davis 
et  al.  (1995)  have  maintained  that  there  are  presently  few 
, well-trained  professionals  and  leaders  focusing  on 
collaborative  initiatives  in  children's  mental  health  service 
delivery.  Additionally,  Duchnowski  and  Friedman  (1990) 
have  discussed  the  need  for  developing  improved 
undergraduate  and  graduate  curriculum  to  train 
professionals  who  will  staff,  organize,  and  manage  these 
emerging  systems  of  care.  Furthermore,  Duchnowski  and 
Kutash  (1995)  have  suggested  that  training  individuals  in 
children's  mental  health  services  should  ideally  be  provided 
within  an  a multidisciplinary,  public  health  perspective. 

It  is,  of  course,  not  enough  to  proclaim  a new  paradigm  shift 
within  the  mental  health  system  without  infusing  this 
re-conceptionalization,  organization,  and  delivery  of  child 
and  adolescent  mental  health  services  into  a comprehensive 
education  and  training  agenda.  This  paper  provides  readers 
with  a summary  of  the  existing  capacity  for  and  potential  to 
expand  education  and  training  opportunities  in  children's 
mental  health  services  through  a public  health  perspective. 
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Method 

A national  survey  was  utilized  to  summarize  the  existing 
capacity  for  and  potential  to  expand  education  and  training 
opportunities  in  children's  mental  health  services  within 
schools  of  public  health.  A two  page,  nine  item, 
self-administered  questionnaire  was  developed  and  sent  to 
all  27  schools  of  public  health  in  the  United  States  and 
Puerto  Rico  which  were  accredited  (as  of  the  fall,  1995)  by 
the  Council  on  Education  in  Public  Health.  The  survey 
instrument  sought  information  on  the  existence,  substantive 
nature,  and  availability  of  mental  health  and  substance 
abuse  courses,  areas  of  concentration,  as  well  as  degree 
programs  within  these  colleges  of  public  health.  It  also 
contained  questions  which  sought  information  on  how  these 
courses  and  programs  were  organized  within  the  schools  of 
public  health.  Additionally,  the  survey  contained  questions 
which  permitted  the  collection  of  information  regarding 
child  and  family  health  courses  which  were  offered  during 
the  1995-1996  academic  year. 

Results 

Mental  Health 

All  27  questionnaires  were  completed  and  returned, 
resulting  in  a 100  percent  response  rate.  Nineteen  of  the  27 
schools  of  public  health  offered  courses  in  mental  health 
services,  with  12  of  these  19  schools  offering  two  or  more 
mental  health  courses  for  the  1995-1996  academic  year. 
Eighteen  of  the  27  schools  of  public  health  offered  courses 
in  social  and  behavioral  sciences. 

A variety  of  mental  health  courses  was  taught  in  the  1 9 
schools  of  public  health.  However,  the  topics  most 
frequently  offered  were  mental  health  epidemiology  (25 
courses  taught  nationally),  mental  health  planning, 
administration,  policy  and  law  (12  courses  taught 
nationally),  mental  health  statistics  and  research  (1 1 courses 
taught  nationally),  children  and  family  mental  health 
services  (eight  courses  taught  nationally),  and  stress  (seven 
courses  taught  nationally).  The  courses  taught  in  children 
and  family  mental  health  within  schools  of  public  health 
included;  Family  and  Mental  Health  (University  of 
California  at  Los  Angeles);  Childhood  Mental  Disorders: 
Public  Health  Perspectives  (Harvard  University);  Children’s 
Mental  Health  (University  of  Illinois  at  Chicago); 
Epidemiology  of  Children's  Psychiatric  Disorders 
(University  of  Pittsburgh);  Mental  Health  of  Children  and 
Adolescents  (The  University  of  Texas);  An  Introduction  to 
Personality  and  Cognitive  Development:  Application  to 
Maternal  and  Child  Health  (Harvard  University);  Child 
Development  and  Psychopathology  (Johns  Hopkins 
University);  and  Public  Health  Interventions  for  Prevalence 
of  Youth  Violence  and  Depression  (University  of  South 
Florida). 
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The  mental  health  courses  were  organized  in  a variety  of 
departments  within  the  schools  of  public  health.  Only  Johns 
Hopkins  University  organized  all  of  the  mental  health 
courses  into  an  independent  departmental  structure 
(Department  of  Mental  Hygiene).  Nevertheless,  15  schools 
of  public  health  contained  health  education,  health  behavior, 
an^or  social  and  behavioral  sciences  departments,  while 
eight  schools  housed  maternal  and  child  health  departments. 

A total  of  nine  schools  of  public  health  offered  either 
master's  degree  programs,  doctoral  degree  programs,  post 
doctoral  degree  programs,  or  areas  of  concentration  in 
mental  health  services  (Loma  Linda  University,  University 
of  California  at  Berkeley,  Yale  University,  University  of 
South  Florida,  University  of  Illinois  at  Chicago,  Harvard 
University,  Johns  Hopkins  University,  Columbia 
University,  and  University  of  Pittsburgh).  Furthermore,  two 
schools  of  public  health  (Johns  Hopkins  University  and 
University  of  Minnesota)  offered  certification  programs  in 
mental  health.  For  the  1995-1996  academic  year,  there  was 
a total  of  37  post  doctoral  fellows  in  mental  health  attending 
seven  schools  of  public  health  (University  of  California  at 
Berkeley,  Yale  University,  University  of  Illinois  at  Chicago, 
Harvard  University,  Johns  Hopkins  University,  Columbia 
University,  and  University  of  Pittsburgh). 

Substance  Abuse 

Substance  abuse  courses  were  offered  by  14  of  the  schools 
of  public  health.  Ten  of  the  14  schools  of  public  health 
offered  two  or  more  substance  abuse  courses  during  the 
1995-1996  academic  year.  Johns  Hopkins  University  and 
the  University  of  Minnesota  offered  five  substance  abuse 
courses,  while  both  the  University  of  California  at  Berkeley 
and  Loma  Linda  University  offered  four  substance  abuse 
courses. 

A total  of  six  schools  of  public  health  offered  areas  of 
concentration,  post  doctoral  programs,  and/or  certification 
programs  in  substance  abuse  (Loma  Linda  University, 
Boston  University,  Harvard  University,  Johns  Hopkins 
University,  University  of  Pittsburgh,  and  University  of 
South  Carolina).  For  the  1995-1996  academic  year,  there 
was  a total  of  seven  post  doctoral  fellows  in  substance  abuse 
attending  two  schools  of  public  health  (Johns  Hopkins 
University  and  the  University  of  Pittsburgh). 

Discussion 

Although  the  first  generation  of  schools  of  public  health  was 
established  prior  to  1920,  almost  one  half  of  the  total 
schools  of  public  health  were  established  and  accredited 
after  1960.  These  interdisciplinary  institutions  have 
significantly  shorter  histories  of  producing  graduate 
students,  vis-a-vis  graduate  programs  in  the  core  disciplines 
of  mental  health  services  (e.g.,  psychiatry;  psychology; 
social  work;  and  psychiatric  nursing).  As  interagency  and 
collaborative  comprehensive  systems  of  care  for  children 
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collaborative  comprehensive  systems  of  care  for  children 
and  their  families  emerge  within  communities,  higher 
education  faces  a tremendous  series  of  both  challenges  and 
opportunities  to  participate  in  the  education  and  training  of 
children's  mental  health  professionals  from  a 
multidisciplinary,  public  health  perspective. 

Public  health,  by  definition,  examines  health  and  disease 
from  multidisciplinary  and  community  perspectives. 
Prevention  and  early  intervention  orientations  are  basic 
foundations  in  public  health.  When  examining  systems 
delivery,  a public  health  framework  incorporates  an 
epidemiologic  perspective,  drawing  on  theories  from 
medicine,  biometry,  environmental  and  occupational  health, 
economics,  sociology,  business,  and  many  other  disciplines. 
Thus,  a public  health  perspective  examines  the  entire  array 
of  problems  faced  by  a specific  population.  Additionally, 
many  schools  of  public  health  have  developed  cross  training 
and  multidisciplinary  (joint)  degree  programs  with  other 
graduate/professional  programs,  including  social  work 
(M.P.H./M.S.W.  programs),  medicine  (M.D./M.P.H. 
programs),  and  law  (J.D./M.P.H.  programs).  Thus,  there 
inherently  exists  within  schools  of  public  health  unique 
opportunities  to  utilize  their  existing  capacity  to  train  mental 
health  professionals  working  with  children  and  their 
families. 

The  results  of  this  study  indicated  that  the  majority  of  the 
schools  of  public  health  (70  percent)  do  not  currently  have  a 
comprehensive  curriculum  and/or  the  expertise  available  to 
train  practitioners  to  work  with  children  and  youth  with 
emotional  disorders.  Historically,  this  lack  of 
multidisciplinary  and  cross-training  of  professionals  can  be 
attributed  to  the  fragmentation  of  the  children's  mental 
health  delivery  systems  that  currently  exists  (Duchnowski 
and  Kutash,  1995). 

Nevertheless,  clearly,  schools  of  public  health  do  have  the 
capacity  to  develop  and  create  expertise  in  both  the  mental 
health  services  delivery  and  the  child  and  family  health 
fields.  Currently,  nearly  three  quarters  (74  %)  of  the 
surveyed  schools  of  public  health  indicated  that  they  offered 
degree  programs,  courses,  post-doctoral  fellowships,  and/or 
continuing  education  opportunities  to  assist  mental  health 
professionals  in  becoming  more  oriented  toward  prevention 
and  early  intervention  paradigms.  Additionally,  44  percent 
of  the  schools  of  public  health  indicated  offering  both 
mental  health  and  substance  abuse  courses. 

The  authors  are  currently  working  with  the  University  of 
South  Florida  College  of  Public  Health  in  developing  an 
area  of  concentration  in  mental  health  and  substance  abuse 
services  at  both  the  master's  and  doctorate  levels.  This  area 
of  concentration  in  mental  health  and  substance  abuse 
would  be  open  to  students  from  any  of  the  four  departments 
within  the  USF  College  of  Public  Health.  Furthermore, 
graduate  students  enrolled  in  the  Doctoral  Studies  in  Child 
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and  Family  Policy  (based  in  the  USF  Department  of  Special 
Education  and  the  USF/FMHI  Department  of  Child  and 
Family  Studies)  would  also  enroll  in  these  mental  health 
courses,  in  order  to  train  future  professionals  in 
collaborative  integrated  systems  of  care  for  children  and 
adolescents. 
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Introduction 

Richard  Briscoe,  Ph.D. 

These  summaries  present  the  research  projects  of  three 
graduate  students  participating  in  the  Multicultural  Mental 
Health  Training  Program  (MMHTP)  at  the  University  of 
South  Florida's,  Florida  Mental  Health  Institute  (FMHI). 
These  students'  work  involved  the  development  of 
evaluation  or  research  projects  with  ethnically  diverse 
minority  communities. 

The  purpose  of  the  MMHTP  is  to  increase  the  number  of 
ethnic  minority  mental  health  professionals  in  the  state.  The 
training  program  provides  students,  university  and  agency 
staff  with  practical  techniques  applicable  to  mental  health 
service  delivery  in  minority  communities  (Briscoe, 

Sedberry,  & Henderson,  1996).  The  National  Institute  of 
Mental  Health  (1991)  advised  that  additional  research  was 
needed  to  support  service  delivery  to  minority  populations. 
MMHTP  collaborates  with  the  Multicultural  Child  and 
Family  Project  (MCFDP)  at  the  FMHI  in  developing 
evaluation,  technical  assistance  and  training  strategies 
aimed  at  improving  the  effectiveness  of  innovative 
community-based  neighborhood  programs  which  serve 
ethnically  diverse  populations  (Briscoe,  Wright,  & Yang, 
1994).  Effective  interventions  with  ethnically  diverse 
communities  address  the  cultural  values  and  unique  needs  of 
local  communities,  because  ethnic  or  racial  groups  differ  in 
their  relevant  cultural  values,  norms,  expectations,  and 
attitudes  (Isaacs  & Benjamin,  1991).  The  MCFDP  and 
MMHTP  embrace  a neighborhood-based  approach  to 
delivery  of  social  services,  and  conceptualize  three 
theoretical  orientations  which  enhance  the  system  of  care 
model's  applicability  to  ethnically  diverse  populations:  (1) 
neighborhood-driven  participation  which  emphasizes 


415 


natural  support  networks  within  the  community  and 
democratic  participation;  (2)  primary  prevention  to  promote 
the  well-being  of  families  and  communities;  and  (3)  cultural 
competence  on  the  part  of  change  agents  to  incorporate  the 
concerns  and  values  of  the  local  residents  into  effective 
programming. 

An  aim  of  the  following  research  projects  was  to  develop 
collaborative  ties  between  ethnically  diverse  communities 
and  the  university  to  increase  the  effectiveness  of  service 
delivery.  The  first  summary  examines  the  strengthening  of 
linkages  between  the  community  and  schools.  National 
educational  goals  recognize  the  importance  of  partnerships 
between  schools  and  the  home  to  the  educational 
achievement  of  children.  How  these  linkages  are  perceived, 
created  and  maintained  in  a predominantly 
African-American  community  was  the  focus  of  this 
dissertation  in  Applied  Anthropology.  The  second  summary 
reports  the  development  of  a community-based  job  services 
program.  This  project  involved  the  coordination  of 
multiagency  family  career  and  job  placement  services  to 
transition  low  income  community  residents  into  improved 
employment  and  self  sufficiency.  The  third  summary,  from 
a public  health  perspective,  addressed  communicating 
health  information  effectively  to  members  of  middle  income 
African-American  communities.  This  project  involved 
promoting  proper  management  of  household  hazardous 
waste  to  improve  indoor  air  quality  and  decrease  the 
prevalence  of  childhood  respiratory  ailments. 
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Working  Together  for  the  Children: 
School-Community  Partnerships  in  East  Tampa 

M.  Yvette  Baber,  M.A. 

Introduction 

This  report  details  the  results  of  a qualitative  evaluation  of  a 
locally-based  effort  to  increase  levels  of  interaction  between 
schools  and  parents  from  East  Tampa,  FL. 

Multicultural  Community  and  Family  Development  Project. 
The  Multicultural  Community  & Family  Development 
Project  (MCFDP)  at  FMHI  has  worked  with  the  East  Tampa 
School-Community  Partnership  (ETSCP)  for  the  past  five 
years  to  stimulate  the  development  of  a coordinated  system 
of  parent-school  partnerships  among  the  elementary  and 
secondary  schools  serving  the  area.  This  internship 
assignment  was  to  provide  evaluation  and  professional 
support  services  to  the  East  Tampa  School-Community 
Partnership  and  to  assess  the  ways  in  which  ETSCP  has 
achieved  its  stated  purpose. 

East  Tampa  School-Community  Partnership.  In  the  late 
1980s,  one  problem  that  was  evident  to  the  Lee  Davis 
Neighborhood  Service  Center  (LDNSC)  and  a parent  from 
an  elementary  school  was  the  disproportionate  level  of 
suspensions  and  expulsions  for  children  who  lived  in  East 
Tampa.  The  East  Tampa  School  Children  Advocate 
Committee  was  formed  in  1990  to  provide  a network  of 
school  and  community  support  to  address  parents'  lack  of 
access  to  their  child's  school,  misconceptions  about  the 
community  or  the  School  District,  and  the  high  levels  of 
disciplinary  actions  experienced  by  the  children  from  East 
Tampa.  In  1992,  the  name  of  the  organization  was  changed 
to  the  East  Tampa  School-Community  Partnership 
(ETSCP).  Each  month  since  then,  principals,  teachers, 
parents,  school  district  staff,  and  social  service  staff  have 
met  to  discuss  these  issues. 

The  stated  purposes  of  the  ETSCP  are:  1)  to  develop  a 
network  of  concerned  citizens  to  bridge  the  communication 
gap  between  home  and  school  by  identifying  obstacles  to 
healthy  communication  in  the  arena  of  education,  and  2)  to 
develop  strategies  to  increase  home-school  communication 
in  the  arena  of  education.  As  the  organization  nears  its  fifth 
full  year  of  operation,  it  is  beginning  to  examine  how  it  has 
reached  its  goals,  with  a focus  on  strengthening  the 
Partnership's  infrastructure  and  increasing  local  parent 
involvement  in  its  activities. 

Evaluation  Methodology 
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Evaluation  Methodology 

The  assessment  of  ETSCP  was  conducted  using  a 
multi-method  approach.  First,  a review  of  the  literature 
related  to  participation  and  school  involvement  in 
African-American  communities  was  completed.  Historical 
documents  were  reviewed,  and  a draft  evaluation  of  the 
ETSCP  (MCFDP,  1 995)  provided  critical  statistical 
information  on  organizational  changes.  This  effort  identified 
key  topics  which  were  discussed,  initiatives  which  were 
undertaken,  and  those  individuals  who  had  consistently 
attended  the  meetings  of  the  organization.  An  intensive 
review  of  the  partnership's  efforts  (January  - December 
1995)  identified  its  accomplishments  for  this  period.  This 
review  produced  a picture  of  an  organization  staffed 
completely  by  volunteers  that  has  consistently  worked  to 
bring  parents  and  teachers  together  for  the  education  of  East 
Tampa's  children. 

The  anthropological  focus  of  the  evaluations  made 
participant  observation  a primary  research  tool.  Intense 
involvement  in  general  membership  meetings  and  board 
meetings  contributed  to  an  understanding  of  the  dynamics  of 
interaction  and  communication  between  members  of  the 
partnership,  the  Hillsborough  county  school  system,  and  the 
East  Tampa  community.  In  addition  to  these  meetings,  the 
author  attended  the  open  parent-school  events  scheduled  at 
the  Lee  Davis  Neighborhood  Service  Center  or  involving 
the  ETSCP  (Back-to  School  Kickoff,  Parent  Enhancement 
Conference,  Christmas  Giving  Tree).  Individuals  who 
belonged  to  the  ETSCP  were  interviewed  about  their 
experiences  and  perceptions.  These  individuals  were 
selected  based  on  their  membership  in  the  organization  and 
their  willingness  to  be  interviewed.  They  consisted  of 
community  residents,  school  district  staff,  principals  and 
assistant  principals  of  partner  schools,  and  social  services 
staff  from  the  Service  Center. 

Evaluation  Findings 

The  review  of  records  indicated  that  the  ETSCP  has,  in  fact, 
drawn  a diverse  group  of  educators,  social  service  staff, 
residents  and  parents  together.  School  and  agency 
information  was  shared  at  virtually  every  meeting,  and 
community  concerns  were  addressed  consistently  over  the 
past  six  years.  From  these  efforts,  vehicles  to  link  schools 
and  community  were  created.  The  Back-to-School  Kickoff 
is  an  annual  event  where  schools  come  to  the  community  to 
meet  and  provide  information  to  parents,  and  where  children 
are  given  school  supplies.  It  has  grown  from  serving  200 
people  in  its  first  year  to  almost  1,000  people  in  1996. 
Additionally,  the  Full  Service  Schools  Program  in  East 
Tampa  was  developed  and  funded  as  a result  of  the  efforts 
of  ETSCP  members  and  is  now  an  integral  source  of 
school-based  services. 

There  are  programs  and  schools  that  have  also  maintained  a 
consistent  relationship  with  ETSCP.  The  public  library,  the 
Tampa  Police  Department,  the  LDNSC,  the  Head  Start 
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Tampa  Police  Department,  the  LDNSC,  the  Head  Start 
Program,  and  the  Tampa  Housing  Authority  have  sent 
representatives  to  meetings  for  the  past  four  or  five  years. 
There  are  some  schools  that  have  demonstrated  a 
commitment  to  linkages  and  working  together  for  the 
children  through  their  regular  participation  and 
involvement.  Nine  Hillsb^orough  County  Schools  have  sent 
representatives  to  meetings  on  a regular  basis  since  1991 . 
These  schools,  and  their  staff,  have  been  key  players  in 
ETSCP's  efforts  to  increase  home-school  communication. 

The  number  of  parents  who  attend  the  monthly  meetings  has 
also  increased  over  the  years,  and  1995  showed  the  most 
impressive  participation.  At  the  beginning  of  the  year,  the 
number  of  parents  varied  from  2-5.  From  July  to  the  end  of 
the  year,  the  numbers  increased,  and  were  in  the  range  of  5 - 
1 6 at  each  meeting.  The  Partnership  is,  indeed,  bringing 
schools  and  parents  together  to  bridge  the  gaps  of 
communication  and  cooperation  that  had,  in  the  past, 
plagued  the  East  Tampa  community. 

The  individual  interviews  revealed  that  the  people  who  have 
joined  the  partnership  are  convinced  that  it  is  a valuable 
organization  and  that  they  are  better  able  to  serve  the 
children  of  East  Tampa  because  of  its  work.  School  district 
personnel  expressed  concern  over  low  levels  of  parent 
involvement  in  the  schools,  but  they  were  able  to  experience 
parent  involvement  at  the  community  level  through 
Partnership  activities.  Parents,  teachers,  and  principals  all 
felt  that  working  together  on  planning  committees  and  at 
ETSCP  events  helped  them  understand  each  other  better. 
Parents  expressed  more  confidence  in  approaching  school 
site  staff  (teachers,  counselors,  etc.)  and  reported  that  they 
appreciated  the  opportunities  they  were  given  to  meet  with 
these  people  in  their  own  community. 

One  factor  that  came  up  consistently  in  meeting  discussions, 
interviews,  and  casual  conversations  was  the  impact  of 
district-wide  bussing  on  the  relationships  between  schools 
and  parents.  Parents  said  that  they  could  not  easily  get  to 
some  of  the  schools  that  their  children  attended  because  of  a 
lack  of  transportation.  Social  services  and  school  staff 
talked  about  the  inability  to  make  a personal  connection 
with  parents  and  the  way  it  affected  classroom  management, 
resolution  of  problems,  and  support  for  the  school  programs. 
All  groups  acknowledged  the  importance  of 
parent-school-community  linkages  but  expressed  frustration 
at  finding  a way  to  build  those  linkages.  They  saw 
community-based  events  as  one  effective  tool  to  draw 
parents  out  and  emphasized  the  need  to  continue  to  reach 
out  to  the  "grass  roots"  in  order  to  build  their  trust  and 
participation. 

Conclusion 

The  success  of  ETSCP  continues  on  its  upward  spiral  - 
drawing  more  people  to  its  general  meeting  and  working 
hard  to  reinforce  the  developing  linkages  between  schools 
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hard  to  reinforce  the  developing  linkages  between  schools 
and  the  community.  The  qualitative  evaluation  of  the 
process  of  achieving  this  goal  has  shown  that  long-term 
volunteer  efforts  can  succeed  with  consistent  participation 
and  a continuing  focus  on  the  purposes  of  the  organization. 
The  Partnership  helps  close  the  gap  between  schools  and 
residents  of  an  inner-city  neighborhood&endash;a  gap 
created  by  forces  beyond  the  control  of  either  the  parent  or 
the  schools.  Parents  and  school  personnel  can  meet  and 
work  together  on  projects  for  the  children  that  are 
community-based.  The  school  personnel  gain  an 
understanding  of  the  environmental  realities  of  their 
students,  and  the  parents  gain  confidence  in  their  ability  to 
communicate  effectively  with  the  people  responsible  for  the 
schooling  of  their  children.  The  Partnership  has  gained 
district-wide  attention  for  its  efforts,  and  discussion  is  now 
underway  about  how  to  develop  similar  partnerships  in 
other  neighborhoods  in  Hillsborough  County,  Florida. 

Community-based  Job  Services  Programs:  The 
First  Step  Toward  Community  Economic 
Revitalization 

Tracy  Lea  McPhail,  M.A. 

Introduction 

This  summary  describes  the  use  of  community-based  job 
development  efforts  to  initiate  economic  revitalization. 

The  issue  of  welfare  reform  and  the  need  to  economically 
revitalize  certain  communities  in  Tampa  prompted  the 
community  leaders  and  concerned  faculty  at  the  University 
of  South  Florida  (USF)  to  address  ways  to  transition  low 
income  welfare  recipients  toward  gainful  employment  and 
self-sufficiency.  The  Center  to  Develop  Communities  of 
Tampa  (CDC  of  Tampa),  formerly  known  as  the  Lee  Davis 
Neighborhood  Development  Corporation  (LDNDC),  has 
been  dedicated  to  developing  and  sustaining  community 
enhancement  programs  for  many  years.  Such  programs 
include  the  Men  to  Boys  program  which  pairs  successful 
African-American  men  with  African-American  boys  who 
are  at  risk  for  developing  problems  in  the  community. 
Another  successful  program  is  the  East  Tampa 
School-Community  Partnership  which  has  been 
instrumental  in  opening  the  lines  of  communication  between 
school  administrators  and  concerned  citizens. 

A 1993  Needs  Assessment  of  East  Tampa  Residents 
revealed  that  the  respondent's  top  three  needs  were  jobs,  day 
care,  and  affordable  single  family  housing.  The  urgency  of 
these  needs  prompted  several  partnerships  between  the 
university  and  community  organizations.  One  such 
partnership  was  initiated  by  Florida  Community 
Opportunity  Partnership  Center  (FCOPC)  and  Multicultural 
Child  and  Family  Development  Project  of  the  Department 
of  Child  and  Family  Studies  at  the  University  of  South 
Florida's,  Florida  Mental  Health  Institute.  These  projects  at 
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Florida's,  Florida  Mental  Health  Institute.  These  projects  at 
the  USF  united  with  the  LDNDC,  the  Department  of  Labor 
and  Employment  Security  (DLES),  and  the  City  of  Tampa 
Private  Industry  Council  (PIC)  to  create  "one  stop" 
comm  unity -based  job  development  centers.  These  facilities 
would  serve  as  a tangible  first  step  in  transitioning  the 
unemployed  toward  self-sufficiency.  One-stop  job 
development  centers  are  designed  to  provide  referrals  for 
jobs,  vocational  training  and  education.  These  centers  are 
comprehensive,  in  that  they  also  address  many  barriers  to 
sustaining  employment  such  as  day  care/elder  care  and 
transportation. 

East  Tampa  is  one  of  four  sites  for  job  development  centers. 
The  Job  and  Education  Placement  Center  (JEPC)  is  housed 
within  the  Lee  Davis  Neighborhood  Development 
Corporation  (LDNDC).  The  JEPC  is  designed  to  address  the 
needs  of  the  unemployed  and  underemployed  residents  in 
the  target  area.  With  the  help  of  the  aforementioned  entities, 
the  JEPC  provides  many  services  to  their  clients.  The  DLES 
has  set  up  several  job  information  services  terminals  which 
clients  use  to  electronically  search  for  jobs  posted  by 
employers  in  four  Florida  counties.  For  clients  who  are 
unsure  of  their  career  path,  the  JEPC  provides  a 
computerized  vocational  assessment  tool  that  prompts  the 
clients  to  answer  questions  concerning  their  skills  and 
interests. 

In  addition  to  referring  clients  to  area  vocational  schools, 
JEPC  also  offers  two  job  training  programs.  Job  readiness 
training  provides  forty  hours  of  customized  instruction  and 
hands-on  practice  designed  according  to  the  employer's 
human  resource  needs.  The  Florida  Aquarium  used  JEPC's 
job  readiness  training  during  its  initial  staffing.  The  JEPC 
also  offers  an  employability  skills  training  program.  This 
program  consists  of  weekly  courses  on  issues  such  as 
resume  writing,  interviewing  skills,  and  proper  business 
attire.  This  training  is  offered  in  several  locations  in  East 
Tampa  including  the  Lee  Davis  Neighborhood  Service 
Center  and  various  housing  developments.  When  training 
participants  appear  to  have  low  self-esteem,  the  staff 
conducts  motivational  exercises  before  presenting  the 
traditional  employability  training  material.  These  exercises 
help  the  program  participants  recognize  their  worth,  while 
identifying  the  skills  they  can  offer  to  prospective 
employers. 

After  all  job  searches  and  job  readiness  evaluations  are 
completed,  the  client  may  be  referred  to  a job  or  a 
vocational  training  facility.  If  job  or  education  placement 
occurs,  then  the  JEPC  should  monitor  their  clients'  progress 
for  90  days.  The  current  staff  capacity  at  JEPC  does  not 
allow  for  comprehensive  monitoring  of  all  clients. 

Program  Evaluation 

Within  its  first  month  of  operation,  JEPC  had  served  90 
clients.  All  the  client  demographic  data  was  not  available. 
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clients.  All  the  client  demographic  data  was  not  available, 
however  simply  comparing  the  responses  of  the  needs 
assessment  to  JEPC's  client's  in  its  first  month  of  operation 
shows  that  JEPC  is  truly  serving  a much  needed  function  in 
the  community. 

Five  months  later,  more  than  350  people  have  yisited  the 
CDC  of  Tampa  office.  Of  those  people,  74  have  been  placed 
in  jobs,  demonstrating  a one  out  of  five  placement  rate.  This 
placement  rate  is  two  placements  higher  than  the  larger  Job 
and  Benefits  Offices.  The  types  of  jobs  vary  from  janitorial 
work  to  mechanical  engineering.  The  types  of  organizations 
that  have  hired  clients  include  GTE  Mobilnet,  Ametek, 

USF,  and  Creative  World  Day  Care. 

Jobs  and  Community  Revitalization 

When  examining  the  community  economics  from  a 
macroeconomic  perspective,  it  is  clear  that  jobs  are  the  first 
steps  toward  community  economic  development. 
Communities  like  East  Tampa  are  constantly  recruiting 
businesses  to  locate  in  their  area.  Businesses  are  attracted  to 
a locale's  amenity  factors  such  as  tax  incentives,  skilled 
workers,  quality  education,  quality  transportation,  and  a low 
crime  rate  (Kasarda  & Irwin,  1991).  East  Tampa  is  fertile 
ground  for  economic  revitalization.  As  a federally 
designated  enterprise  zone,  businesses  located  within  the 
target  area  receive  tax  incentives  and  other  perks. 

Community-based  job  centers  are  a crucial  link  in 
improving  the  amenity  factors  that  can  attract  businesses  to 
the  area.  As  businesses  use  the  JEPC  to  find  qualified, 
responsible  employees,  the  residents  acquire  income  that 
can  be  spent  within  the  community.  Continued  efforts  to 
i recruit  more  businesses  in  the  community  will  help 
I diversify  the  local  industry  mix.  The  revenue  from  goods 
I and  services  consumed  (and  eventually  produced)  in  East 
Tampa  will  contribute  to  the  community's  economic 
expansion.  The  process  all  begins  with  jobs  (Hindley,  1981; 
Kasarda  & Irwin,  1991). 
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One  of  the  primary  goals  of  this  study  was  to  determine  the 
preferred  sources  of  health  information  for  middle-income 
African-American  families,  as  well  as  how  health  risks  are 
perceived  differently  among  ethnic  populations.  The  health 
risk  example  identified  for  this  study  was  exposure  to 
environmental  contaminants,  as  the  condition  and  quality  of 
one's  surrounding  environment  can  have  significant 
influence  on  physical,  mental,  and  emotional  health.  The 
most  common  exposure  to  environmental  contaminants 
occurs  in  our  homes  with  common  household  products  that 
contain  hazardous  chemicals.  This  summary  describes  this 
exploratory  research's  methodological  procedures  and 
preliminary  implications. 

Methods 

This  study's  selected  target  population,  middle-income 
African  Americans,  is  one  of  the  fastest  growing  economic 
sectors  in  American  society.  Currently,  three  out  of  ten 
African-American  families  in  the  United  States  have 
attained  middle-income  socioeconomic  status.  Conveying 
health  information  effectively  to  these  families  should  be  a 
primary  concern  for  prevention.  To  be  eligible  for  this 
study,  participants  had  to  reside  in  Hillsborough  County, 
live  in  a single-family  dwelling,  and  have  at  least  one  child 
under  the  age  of  ten  years  old. 

This  study  used  direct  observations,  elite  interviews,  focus 
groups,  and  community  surveys  to  gain  information  on 
perceptions  and  preferred  sources  of  health  information,  as 
well  as  more  specific  concerns  and  behaviors  related  to 
environmental  health  Fifteen  direct  observations  estimated 
product  preferences,  usage  levels,  and  storage  methods. 
Thirty  interviews  with  community  leaders  provided  insight 
about  the  target  population;  quotes  from  focus  groups  also 
were  instrumental  in  designing  preliminary  health  messages 
using  the  cultural  cues  and  values  from  middle-income 
African-American  parents.  Results  of  these  qualitative 
research  methods  reflected  a range  of  opinions  and 
concerns,  and  proved  beneficial  to  preliminary  development 
of  community  surveys.  These  surveys  were  distributed  to  a 
sample  of  250  homes  from  six  selected  neighborhoods  in 
Hillsborough  County.  The  selected  neighborhoods  were 
comprised  of  at  least  50%  African-American  residents, 
based  on  1990  Census  data.  At  least  40%  of  the  households 
had  children  present,  and  at  least  40%  of  the  households 
earn  more  than  $25,000  per  year. 

So  how  does  this  study  translate  into  practical  application? 
Preliminary  analysis  suggests: 

1 . With  a limited  number  of  ethnic  populations  utilizing 
available  mental  health  services,  it  is  imperative  to 
address  current  health  concerns  in  the  marketing  of 
current  resources  through  a focus  on  the  community 
strengths. 

2.  Cultural  cues  and  pivotal  messages  may  be  slightly 
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more  distinct  for  such  a specific  demographic 
population.  Thus,  quotes  from  interviews  and  surveys 
may  serve  to  peak  the  interest  when  describing 
available  programs. 

3.  Due  to  this  population's  typically  strong  and  constant 
spiritual  beliefs,  an  avenue  of  joining  forces  with  a 
particular  religion's  or  denomination's  health 
ministries  may  be  explored  further. 

Environmental  Psychology  Implications 

On  another  level,  perceptions  of  the  environment  or 
surroundings  may  yield  a reflection  of  mental  health 
influences.  Nature  or  environmental  experiences  can  help 
broaden  our  sense  of  the  world  while  deepening  our  sense  of 
ourselves  (Wals,  1994).  Qualitative  and  quantitative 
findings  suggest  that  those  residing  in  urban  ethnic 
communities,  without  regard  to  socioeconomic  status,  are 
often  exposed  to  excessive  air,  water,  and  soil  pollution.  A 
common  response  is  concern  for  the  cumulative  impact  on 
their  children's  health  twenty  years  from  now.  Door-to-door 
community  surveys  revealed  shared  concerns  and  emotions 
that  require  solution  to  ensure  the  stability  of  future  physical 
and  mental  health. 

Feedback 

The  community  survey  data,  complemented  with  qualitative 
approaches,  will  provide  concrete  suggestions  and  strategies 
for  addressing  a distinct  ethnic  population  with  multiple 
community  networks  throughout  the  socioeconomic  scale 
within  the  black  community.  Feedback  to  the  community  is 
a critical  component  of  any  research.  In  this  instance,  the 
community  will  receive  information  in  the  form  of  data  that 
can  be  transformed  to  program  ideas,  radio  talk  shows, 
newspaper  articles,  and  meeting  presentations.  Based  on  the 
information  and  concerns  expressed,  an  environmental 
education  program  can  be  designed  to  include  topics  of 
most  interest  to  the  communities,  increasing  adults 
awareness  of  the  impact  of  environmental  contaminants  on 
the  family's  physical,  mental,  and  psychological  health. 
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Conclusion 

This  symposium  was  intended  to  highlight  the  research 
projects  of  students  participating  in  the  MMHTP  in 
coordination  with  the  MCFDP.  These  research  projects 
address  the  overall  objective  of  the  MMHTP  to  improve  the 
quality  of  services  to  minority  communities  through  a 
combination  of  training,  research  and  treatment  services. 
These  minority  student  researchers  are  dedicated  to 
understanding  and  studying  the  mental  health  needs  of  an 
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increasingly  ethnically  diverse  population.  Initially,  the 
MMHTP  placed  students  in  community  sites  to  offer 
treatment  services  and  technical  assistance.  These  current 
students  are  combining  research  and  evaluation  procedures 
to  improve  service  delivery  within  communities.  Data 
collected  in  these  projects  involved  community-based 
programs  or  community  residents.  The  data  gathered 
required  formal  and  informal  linkages  between  community 
residents  and  the  researchers.  The  roles  of  these  researcher 
are  viewed  as  collaborators  or  partners  for  successful  service 
delivery.  The  MMHTP  will  continue  to  offer  minority 
students  the  opportunity  to  develop  research  projects  and 
collaborative  approaches  for  improving  service  delivery 
with  ethnically  diverse  populations. 
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Introduction 

Given  the  current  fiscal  environments  in  which  State  Mental 
Health  Administrations  are  operating,  there  is  increasing 
pressure  to  develop  empirical  methods  of  assessing  need 
and  allocating  resources.  In  1995,  the  Virginia  Department 
of  Mental  Health,  Mental  Retardation,  and  Substance  Abuse 
Services  (DMHMRSAS)  conducted  a study  to  evaluate  the 
need  for  services  across  the  continuum  of  child/adolescent 
mental  health  services.  The  specific  objectives  of  this  study 
were  to:  (a)  describe  the  youth  receiving  services,  (b) 
determine  where  and  to  what  extent  gaps  in  services  exist, 

(c)  determine  the  reasons  why  youth  do  not  receive  services, 

(d)  empirically  classify  youth  on  the  basis  of  "service 
packages"  received  (or  identified  as  needed),  and  (e) 
identify  those  sociodemographic,  clinical,  and  treatment 
history  variables  that  best  predict  the  service  packages 
identified  as  needed. 


Method 

Data  were  gathered  through  a survey  of  community-based 
case  managers  (or  primary  clinicians)  on  a stratified  random 
sample  of  youth  who  were  receiving  public  mental  health 
services  on  December  3 1 , 1994  (N  = 2,059).  This 
population  was  divided  into  two  distinct  sub-populations, 
youth  with  serious  emotional  disturbance  (SED;  N = 1,100) 
and  non-SED  (N  = 959)  to  determine  whether  their  service 
needs  differed. 

Respondents  to  the  survey  provided  (a)  descriptive  data 
about  the  youth,  (b)  the  types  and  amounts  of  services 
received  during  the  last  quarter  of  1 994,  (c)  the  types  and 
amounts  of  services  that  would  have  been  needed  to  most 
appropriately  and  effectively  serve  the  youth  during  the 
quarter  as  estimated  by  the  youth's  case  manager  or 
clinician,  and  (d)  reasons  for  differences  between  services 
received  and  needed  (e.g.,  service  not  available,  insufficient 
service  capacity,  etc.).  Gaps  in  services  were  determined  by 
calculating  the  difference  between  the  types  and  amounts  of 
services  provided  and  the  types  and  amounts  of  services 
needed. 

Using  cluster  analysis,  "typical"  service  packages  were 
identified,  based  on  services. recommended  as  needed  by  the 
case  managers/clinicians.  The  youth  were  then  classified 
according  to  cluster  membership  and  a discriminant 
function  analysis  was  conducted  using  sociodemographic, 
clinical,  and  treatment  history  variables  to  determine  which 
variables  best  predicted  recommended  service  packages. 

Demographic  variables  that  were  included  in  the 
discriminant  function  analysis  included  age,  race,  gender, 
educational  level,  special  education  enrollment,  residence  in 
private  household,  legal  status  (voluntary  vs.  involuntary), 
annual  income,  SSDl  eligibility,  SSI  eligibility,  and 
payment  source.  Clinical  variables  used  in  the  analysis 
included  number  of  prior  episodes,  history  of  previous 
psychiatric  hospital  admissions,  SED  status,  and  score  on 
the  Global  Assessment  of  Functioning  (GAF).  Due  to  the 
low  frequency  of  some  of  the  diagnostic  categories  used  in 
the  survey,  primary  and  secondary  diagnoses  were  grouped 
into  the  following  categories:  (a)  schizophrenia  and  other 
psychoses,  (b)  adjustment  disorders,  (c)  mental  retardation 
or  pervasive  developmental  disorder,  (d)  attention-deficit  or 
disruptive  behavior  disorders,  (e)  anxiety  or  mood  disorder, 
(f)  substance  dependence  or  abuse,  and  (g)  other. 

Results 

Consumer  Profile 

The  typical  youth  enrolled  for  services  at  a Community 
Mental  Health  Center  (CMHC)  in  Virginia  was  a male 
(60%),  13  to  17  years  of  age  (50%).  African-American 
youth  were  over-represented  relative  to  their  percentage  in 
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youth  were  over-represented  relative  to  their  percentage  in 
the  general  population  (29%).  Half  of  the  youth  with  SED 
were  enrolled  in  special  education.  The  majority  of  these 
youth  (83%)  came  from  families  with  total  annual  incomes 
of  less  than  $20,000.  Medicaid  was  the  most  common 
source  of  payment  for  CMHC  services,  although  the 
consumer's  family  was  the  direct  source  of  payment  for  a 
large  percentage  (29%)  of  youth  without  SED. 

Among  youth  with  SED,  primary  diagnoses  of  disruptive 
behavior,  attention  deficit,  anxiety,  and  mood  disorders 
were  most  common,  accounting  for  83%  of  the  youth.  By 
contrast,  among  consumers  without  SED,  the  most  common 
primary  diagnosis  was  adjustment  disorder  (27%). 

Disruptive  behavior,  attention  deficit,  and  mood  disorders 
were  also  very  common,  accounting  for  an  additional  44% 
of  youth  without  SED. 

Service  Needs 

In  order  to  identify  gaps  in  service  capacity,  the  total 
amount  of  individual  services  recommended  was  compared 
to  the  total  amount  received.  The  CMHCs  were  able  to  meet 
a substantial  amount  of  estimated  need  for  inpatient  and 
medication  management  for  consumers  with  SED  (greater 
than  90%  and  80%,  respectively).  For  consumers  without 
SED,  estimated  need  for  inpatient  and  highly  intensive 
residential  services  was  most  frequently  met  (greater  than 
70%).  With  respect  to  gaps,  the  largest  gaps  for  youth  with 
SED  were  found  for  in-home  supported  living,  day 
treatment,  and  residential  support  where  less  than  50%  of 
recommended  services  were  received.  For  youth  without 
SED,.the  largest  gaps  were  found  for  residential  support, 
in-home/supported  living,  rehabilitation,  day  treatment,  and 
early  intervention;  less  than  30%  of  the  total  estimated 
services  needed  were  provided. 

Reasons  for  Gaps 

The  reasons  for  insufficient  amounts  of  services  being 
provided  were  consistent  regardless  of  SED  status.  "Lack  of 
capacity"  was  the  reason  reported  for  the  greatest  percentage 
of  youth  not  receiving  needed  services.  "Missed/no  show" 
and  "consumer/family  decision"  were  also  significant 
factors,  particularly  for  outpatient  and  medication 
management  services.  For  this  latter  service  category, 
"consumer/family  decision"  was  the  most  frequently 
occurring  reason.  Other  reasons,  cited  only  rarely,  were 
"consumer  behavioral  problems,"  "insufficient  consumer 
resources,"  and  "inaccessibility." 

Cluster  Analysis 

The  cluster  analysis  was  limited  to  consumers  who 
responded  to  the  Virginia  Service  Assessment  Questionnaire 
and  Continuum  of  Care  Consumer  Profile.  The  final  data  set 
included  1097  youth  with  SED  and  799  non-SED  youth. 
Because  the  number  of  youth  with  SED  were  not  in 
proportion  to  the  number  of  non-SED  youth,  the  non-SED 
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proportion  to  the  number  of  non-SED  youth,  the  non-SED 
sample  was  weighted  by  1 .67.  This  weight  was  derived 
from  the  percentage  of  SED  cases  in  the  sample  weighted 
by  geographic  location.  This  resulted  in  a total  sample  for 
the  cluster  analysis  of  2428  youth. 

In  order  to  classify  consumers  according  to  the  service 
packages  recommended  by  case  managers,  a cluster  analysis 
using  the  K-means  procedure  of  the  Statistical  Package  for 
Social  Sciences  (SPSS)  was  conducted.  Cluster  solutions 
using  3,  4,  and  5 clusters  were  computed  in  an  effort  to 
identify  the  solution  which  created  clusters  with  sufficient 
sample  sizes  (N  > 30)  to  allow  for  further  data  analysis  and 
ease  in  interpretation.  Using  this  criterion,  the  four  cluster 
solution  was  chosen  as  the  preferred  solution  for  services 
recommended. 

Using  cluster  membership  as  the  independent  variable  and 
number  of  service  hours  per  service  type  as  the  dependent 
variable,  a one-way  analysis  of  variance  (ANOVA)  was 
used  to  identify  those  service  types  that  distinguished 
among  the  clusters.  When  the  F-score  for  a given  service 
type  was  found  to  be  statistically  significant,  the 
Student-Newman-Keuls  procedure  was  used  to  differentiate 
among  clusters  on  that  service  (see  Table  1). 

To  interpret  the  clusters,  attention  was  given  to  those 
services  shown  by  the  post-hoc  analysis  to  most  clearly 
distinguish  each  cluster  from  the  other  clusters.  Individuals 
in  Cluster  1,  labeled  Residential  Services,  needed  the  most 
units  of  intensive  residential  services  and  outpatient 
services.  The  most  characteristic  feature  of  Cluster  2,  which 
described  over  90%  of  the  sample,  was  the  low  rate  of  all 
services  recommended.  Therefore,  this  group  was  labeled 
Minimal  Services.  Individuals  in  Cluster  3,  labeled  Inpatient 
Services,  were  described  as  needing  the  highest  rates  of 
inpatient,  residential,  and  case  management  services. 
Individuals  in  Cluster  4 were  recommended  for  the  widest 
variety  of  services  with  need  for  residential  support,  day 
treatment,  and  vocational  services  exceeding  that  of  the 
other  groups.  These  services  are  primarily  community  based 
and  of  moderate  intensity,  and  therefore,  this  cluster  was 
labeled  Wraparound  Services.  The  stability  of  the  cluster 
solution  was  confirmed  by  selecting  random  subsamples 
and  repeating  the  cluster  procedure  on  the  subsamples  to 
test  for  consistency  of  the  cluster  memberships  between  the 
full  sample  and  subsamples  using  the  Kappa  statistic. 

Discriminant  Function  Analysis 

To  determine  which  demographic  variables  could  predict 
cluster  membership,  a discriminant  function  analysis  was 
conducted  using  the  Wilk's  lambda  method  to  test  for  the 
discriminating  power  of  the  variables  in  the  function.  Given 
the  absence  of  a priori  knowledge  about  the  distribution  of 
group  membership,  the  equal  prior  probabilities  method  was 
used.  The  variables  that  predicted  cluster  group  membership 
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based  on  services  needed  are  presented  in  Table  2. 
Classification  of  cases  to  clusters  based  on  these  variables 
made  61%  fewer  errors  than  would  be  expected  by  chance. 
In  order  to  determine  the  relationship  of  these  variables  with 
cluster  membership,  a one-way  analysis  of  variance  was 
conducted  using  cluster  membership  as  the  independent 
variable  and  the  predictor  variables  as  the  dependent 
variables.  All  of  the  predictor  variables  were  significant  (p 
<.01).  Finally,  the  Student-Newman-Keuls  test  (p  < .05) 
was  used  to  determine  the  significance  of  differences 
between  the  group  means. 

These  findings  indicate  that  children  perceived  as  needing 
Inpatient  Services  (Cluster  3)  are  more  likely  to  be  in 
special  education,  have  a thought  disorder  diagnosis,  have  a 
history  of  previous  mental  health  hospitalization,  and  have  a 
low  GAF  score,  indicating  high  impairment.  Youth  needing 
Residential  Services  (Cluster  1)  were  more  likely  to  have  an 
involuntary  referral  from  the  criminal  justice  system,  a 
diagnosis  of  substance  abuse,  and  a relatively  high  GAF 
score,  indicating  low  impairment.  Non-SED  children  living 
in  a private  home  were  most  likely  to  be  perceived  as 
needing  Minimal  Services  (Cluster  2).  There  were  no  clear 
predictors  of  membership  in  Cluster  4,  Wraparound 
Services. 

Discussion 

Overall,  the  results  of  this  study  indicate  that  in  Virginia's 
public  mental  health  system: 

• The  greatest  needs  are  for  increased  service  capacity 
for  residential  and  day  support  services. 

• In  the  opinion  of  direct  service  staff,  lack  of  service 
capacity  is  the  major  reason  why  consumers  do  not 
receive  the  amounts  of  service  needed. 

• Youth  who  are  recommended  by  clinicians/case 
managers  to  receive  more  intensive  services  fall  into 
three  groups  based  on  typical  service  packages 
recommended  (i.e.,  inpatient,  residential,  and 
wraparound). 

• Several  demographic  and  mental  health  variables  can 
be  used  to  predict  which  youth  will  be  most  likely  to 
use  a particular  "service  package."  These  variables 
include  score  on  the  GAF,  history  of  mental  health 
hospitalization,  diagnosis,  SED  status,  special 
education  status,  legal  status,  and  residence  in  a 
private  home  versus  other  living  arrangement. 

The  results  of  this  study  provide  documentation  of  the 
greatest  service  needs  for  child/adolescent  consumers  of 
public  mental  health  services  within  one  state  system.  There 
appears  to  be  a large  unmet  need  for  all  types  of 
community-based  services,  and  a need  for  CMHCs  to  be 
able  to  offer  a broader  array  of  services  than  they  currently 
offer.  The  results  of  this  study  are  being  used  to  support  and 
develop  a "reinvestment  strategy"  (i.e.,  moving  funds  from 
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inpatient  to  community-based  services)  as  well  as  managed 
care-type  initiatives. 

The  youth  typologies  and  related  service  packages  can 
provide  a useful  tool  for  service  systems  planning. 
Knowledge  of  appropriate  service  packages  for  different 
types  of  youth  will  allow  planners  to  "size"  the  system  of 
care  based  on  the  known  characteristics  of  the  consumer 
population.  Since  there  are  differences  in  the  variables  that 
predict  service  usage  between  youth  and  adults,  examining 
these  populations  separately  is  likely  to  result  in  more 
accurate  estimations  of  service  utilization. 
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This  summary  reports  on  efforts  underway  in  New  York 
State  to  develop  the  Individualized  Needs  for  Services 
Assessment  (INSA),  a set  of  standardized  procedures  and 
data  definitions  to  guide  assessment  of  service  needs  for 
children  with  serious  emotional  disturbances  (SED).  The 
INSA  is  intended  to  be  used  by:  (a)  service  providers  and 
family  members  who  want  to  develop  individualized  service 
plans;  (b)  planners  and  policy  makers  who  want  to  quantify 
need  for  services  at  the  community  level,  and  services 
researchers  who  want  to  study  factors  correlated  with 
service  needs;  and  (c)  managed  care  organizations  and 
purchasers  of  services  who  want  to  balance  issues  of 
ensuring  access  to  needed  services  while  reducing  provision 
of  unnecessary,  ineffective,  or  overpriced  services. 
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A major  challenge  confronting  the  mental  health  field  is  to 
operationalize  and  quantify  the  concepts  of  need  and  unmet 
need  for  services  (Klerman,  Olfson,  Leon,  & Weissman, 
1992;  Bebbington,  1990).  Policymakers,  service  providers, 
and  advocates  for  children  with  emotional  disturbances 
consistently  place  a high  priority  on  the  provision  of 
sufficient  services  to  meet  what  are  perceived  to  be  high 
unmet  needs  for  services.  Methods  for  detecting  and 
diagnosing  emotional  disorders  in  children  have  improved 
and  become  more  standardized.  However,  there  are  no 
standardized  procedures  either  for  determining  which 
services  are  needed  in  individual  cases  or  for  specifying  the 
community  service  capacities  needed  to  serve  diagnostically 
defined  subpopulations. 

Assessment  of  service  needs  is  complicated  by  the  fact  that 
children  with  SED  frequently  are  served  by  multiple 
provider  systems.  Needs  typically  have  been  defined  in 
terms  of  organizational  settings  or  program  components, 
such  as  residential  treatment  facilities,  clinics,  or  day 
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treatment  programs  (Bums,  Angold,  & Costello,  1992; 

Fires,  1990).  Yet,  a program  component  approach  does  not 
clarify  what  service  items  are  needed  and  makes  it  difficult 
to  measure  the  extent  to  which  service  needs  are  met  by 
other  systems  of  care  that  may  deliver  similar  services  but 
use  incompatible  terminology.  It  also  creates  problems  in 
applying  findings  from  one  geographic  region  to  another, 
where  services  may  be  organized  differently,  or  for 
designing  new  forms  of  service  organization  that  would  be 
more  efficient  in  meeting  needs. 

Individualized  Service  Planning 

The  federal  Child  and  Adolescent  Service  System  Program 
(CASSP)  initiative  identified  children  with  SED  as  a high 
priority,  given  that  many  of  them  are  underserved  and  in 
need  of  advocacy,  while  others  are  treated,  often 
ineffectively,  in  costly  inpatient  settings  and  other 
out-of-home  placements  (Stroul  & Friedman,  1986). 

CASSP  selected  the  following  as  basic  parameters  for 
defining  the  population  of  children  with  SED:  (a)  impaired 
social  functioning  in  family,  school,  and  neighborhood 
contexts;  (b)  in  need  of  a range  of  services  requiring  the 
involvement  of  multiple  agencies  including  the  mental 
health,  health,  education,  child  welfare,  juvenile  justice 
sectors,  and  others;  and  (c)  impairment  that  has  lasted  or  is 
expected  to  last  one  year  or  more.  CASSP  envisioned  a 
child-centered,  community-based  system,  providing  a 
comprehensive  array  of  services  including  mental  health, 
social,  educational,  health,  vocational,  recreational,  and 
operational  components  (Stroul  & Friedman,  1986). 

Recent  literature  has  placed  emphasis  on  principles  that  lay 
a foundation  for  services  that  are  (a)  flexible  and 
accommodate  the  individual  needs  of  the  child  and  family, 

(b)  family-focused  and  involve  the  family  in  planning,  and 

(c)  competent  and  meet  the  needs  of  multi-cultural 
populations  (Duchnowski  & Friedman,  1990;  Duchnowski 
& Kutash,  1993;  Katz-Leavy,  Lourie,  Stroul,  & 
Zeigler-Dendy,  1992).  CASSP  envisions  the  development 
of  individualized  service  plans  as  a critical  aspect  of  practice 
within  a comprehensive  system  of  care  and  that  should  have 
a strong  bearing  on  what  services  a child  and  family 
ultimately  receive  (Burchard  & Clarke,  1990;  Duchnowski 
& Friedman,  1990;  Duchnowski  & Kutash,  1993; 
Katz-Leavy  et  al.,  1992;  Rivard,  Perry,  & Hinkle,  1994). 
With  the  growing  programmatic  emphasis  on  individualized 
service  approaches  &emdash;  based  on  the  principles  of 
unconditional  care,  family  involvement,  multi -cultural 
competence,  delivery  in  a child's  natural  environment,  and 
tailoring  to  the  individual  needs  of  the  child  and  family 
&emdash;  knowledge  about  appropriate  assessment  of 
individual  service  needs  is  sorely  needed. 

The  growing  emphasis  on  managed  care  to  control  access  to 
and  coordinate  services  places  additional  importance  on  the 
development  of  a consensus  within  the  child  mental  health 


field  of  what  constitutes  service  need.  Under  traditional 
financing  arrangements  and  multiple,  uncoordinated 
delivery  systems,  there  was  little  incentive  to  reduce  costly 
or  inappropriate  provision  of  services  for  children  who  had 
access.  The  broad  coverage  and  capitation  financing 
arrangements  envisioned  under  managed  care  will  provide 
strong  incentives  to  limit  or  deny  care  when  services  are 
judged  as  unnecessary.  In  the  absence  of  guidelines  for  what 
constitutes  need,  the  impact  of  managed  care  on  children 
with  SED  may  place  children  at  even  greater  risk  for 
under-provision  of  services. 

MRC  Needs  for  Care  Assessment  Approach 

While  the  Medical  Research  Council  (MRC)  Needs  for  Care 
Assessment  Procedure  was  developed  for  adults  with 
psychiatric  disabilities  (Brewin,  Wing,  Mangen,  Brugha,  & 
MacCarthy,  1987;  Mangen  & Brewin,  1991;  Brewin,  1992), 
it  provides  concepts,  procedures,  and  a logic  structure  that 
we  have  adapted  for  assessing  the  service  needs  of  children 
with  SED.  Needs  are  defined  according  to  an  assessment  of 
functioning  that  includes  domains  related  to  psychiatric 
symptoms  and  social  role  performance.  Generic  service 
items  (defined  independent  of  program  components)  are 
listed  under  each  functional  domain  where  they  are 
considered  likely  to  be  effective. 

For  any  particular  individual,  need  is  defined  according  to 
assessment  of  functioning,  the  effectiveness  that  a particular 
service  item  is  expected  to  have  for  this  particular 
individual,  and  the  acceptability  of  the  service  item  to  the 
individual.  The  procedure  includes  a logic  structure  that 
leads  from  judgments  in  these  three  areas  to  categories  of 
need  for  each  domain  and  each  service  item.  In  addition  to 
results  ranging  from  unmet  need,  partially  met  need,  and 
met  need,  the  procedure  allows  for  ratings  of  no  meetable 
need  (i.e.,  when  no  interventions  are  judged  to  be  effective 
and  acceptable  to  address  a domain  problem  for  a particular 
individual)  and  overprovision  (i.e.,  when  the  frequency  or 
intensity  of  a service  item  exceeds  what  is  required  to 
achieve  functional  outcomes). 
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Process 

The  INSA  procedure  for  children  with  SED  is  designed  to 
be  consistent  with  principles  from  CASSP  and 
individualized  service  plaiming  and  applies  data  standards 
and  a logic  structure  adapted  from  the  MRC  Needs  for  Care 
Assessment.  The  INSA  is  being  further  specified  and  tested 
in  The  SED  Study,  an  National  Institute  of  Mental  Health 
(NIMH)-funded  child  mental  health  services  research  study 
being  conducted  by  Columbia  University  (Christina  Hoven, 
Principal  Investigator)  in  Westchester  County  as  well  as  in 
The  FRIENDS  Project,  a Center  for  Mental  Health  Services 
(CMHS)-funded  comprehensive  services  demonstration  in 
the  Mott  Haven  section  of  the  Bronx. 

Interdisciplinary  Teams  with  Family  Participation 

In  both  settings,  service  needs  are  being  assessed  using 
interdisciplinary  teams  with  the  participation  of  family 
members.  In  Mott  Haven,  for  example,  once  a child  is  found 
eligible  for  FRIENDS  (based  on  criteria  for  SED),  a team  is 
assembled  to  develop  an  individualized  service  plan.  Team 
members  include  parents  or  surrogate  parents,  the  child  (as 
appropriate),  and  representatives  from  the  mental  health  and 
education  systems.  Representatives  from  child  welfare, 
juvenile  justice,  alcohol  substance  abuse,  primary  health 
care,  recreation,  and  other  relevant  programs  or  community 
organizations  also  may  be  included.  Information  about 
providers  previously  or  currently  serving  the  family  or 
expected  to  serve  the  family  in  the  future  is  considered  in 
' selecting  team  members. 

The  team  reviews  the  thorough  clinical  and  functional 
assessment  conducted  as  part  of  the  FRIENDS  admission 
process,  along  with  historical  records,  and  gathers  any 
additional  information  required  for  the  needs  assessment. 
Team  members  complete  INSA  forms  individually,  and  then 
the  team  discusses  each  component  of  the  assessment  with 
the  goal  of  achieving  consensus  on  all  phases.  The  team's 
INSA  results  represent  what  the  child  needs  ideally.  This 
information  is  then  used  by  the  team  to  develop  an 
individualized  service  plan  that  factors  in  how  services  can 
best  be  delivered  given  real-world  constraints  of 
availability,  organization,  and  financing  of  existing  services. 
Analyses  comparing  INSA  results  with  individualized 
service  plans  will  be  conducted  to  support  community 
planning  and  funding  allocation.  For  the  Westchester  study, 
a similar  process  is  being  undertaken,  incorporating  an 
expert,  multi-disciplinary  panel  into  the  research  process. 

Data  Standards  and  Logic  Structure 

Functional  Domains 

Functioning  is  assessed  for  the  following  domains;  (1) 
self-care;  (2)  family  life;  (3)  social  and  interpersonal 
relationships;  (4)  learning,  school  performance,  vocational 
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relationships;  (4)  learning,  school  performance,  vocational 
development;  (5)  disruptive  behavior;  (6)  mood  symptoms; 
(7)  anxiety  symptoms;  (8)  symptoms  of  psychosis;  (9) 
attention  deficit  and/or  hyperactivity  symptoms;  and  (10) 
alcohol  and/or  other  substance  abuse.  The  child's 
functioning  in  each  domain  is  rated  according  to  a 
four-point  scale  from  no  problem  to  severe.  Additional 
instrumentation  is  in  development  to  support  functional 
ratings  responsive  to  the  child's  developmental  stage. 

Service  Items 

The  INSA  approach  incorporates  a taxonomy  of  over  75 
generic  service  items  thought  to  be  capable  of  meeting  need. 
"Generic"  refers  to  service  items  that  are  generalizable  and 
comprehensible  across  different  organizational  units  and 
systems  of  care.  Examples  include: 

• psychotherapy,  brief  or  short-term  individual,  for  the 
child; 

• home-based  training  in  parenting  skills,  child 
behavior,  and  symptom  management  for  parent(s)  or 
family  members;  and 

• peer  support,  self-help,  or  support  group  for  the  child. 

For  each  functional  domain,  service  items  thought  likely  to 
be  effective  according  to  expert  judges  are  listed.  Some 
service  items  appear  under  multiple  domains.  For  those 
functional  domains  where  the  child  is  experiencing 
problems,  team  members  rate  service  provision  according  to 
a scale  that  takes  into  account  both  current  and  past  receipt 
of  the  service  item. 

Anticipated  Clinical  Effectiveness 

The  team  rates  the  anticipated  clinical  effectiveness  of  each 
service  item  in  addressing  problems  the  child  is 
experiencing  in  a given  domain  according  to  the  following 
scale:  (0)  demonstrated  ineffective  with  adequate  trial;  (1) 
no  adequate  trial,  but  judged  to  be  ineffective;  (2)  no 
adequate  trial,  but  believed  to  be  effective  or  partly 
effective;  (3)  partly  effective  based  on  adequate  trial;  (4) 
demonstrated  effective  based  on  adequate  trial;  (8) 
inconclusive,  judgment  deferred;  (9)  not  applicable.  These 
judgments  incorporate  knowledge  found  in  the  professional 
literature  (e.g.,  outcome  studies,  practice  guidelines)  as  well 
as  factors  specific  to  the  individual,  such  as  service  history. 

Child  and  Family  Acceptability  of  Services 

The  team  rates  the  level  of  acceptability  of  the  service  item 
to  the  child  and  family  according  to  the  following  scale:  (0) 
rejection;  (1)  not  likely  to  be  acceptable;  (2)  likely  to  be 
acceptable;  (3)  demonstrated  to  be  acceptable;  (8)  uncertain; 
(9)  not  applicable.  This  rating  is  made  considering  the 
family's  prior  experience  with  similar  approaches,  cultural 
appropriateness,  and  other  factors  related  to  individual 
preferences. 
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Need  Status 

Need  status  is  measured  according  to  the  following  scale: 

(0)  no  need,  (1)  no  meetable  need;  (2)  unmet  need  for 
assessment/trial;  (3)  unmet  need  for  provision;  (4)  met  need, 
maintain  current  frequency/intensity;  (5)  met  need,  increase 
frequency/intensity;  (6)  met  need,  assess  overprovision.  In 
addition,  a judgment  as  to  the  presence  or  absence  of 
overprovision  is  made  when  indicated.  A logic  structure  that 
considers  the  ratings  made  on  the  above  dimensions  either 
determines  the  rating  on  need  status  or  narrows  the  response 
categories  available  for  any  given  rating.  For  example,  a 
service  item  with  an  acceptability  rating  of  rejection  or  a 
clinical  effectiveness  rating  of  ineffective  would  lead  to  a 
need  status  of  no  meetable  need.  A computerized  version  of 
the  needs  assessment,  that  automates  the  logic  structure,  is 
being  planned. 

Discussion 

The  INS  A approach  offers  a standardized  set  of  methods  for 
assessing  the  service  needs  of  children  with  SED  that  may 
be  applied  to  individualized  treatment  planning,  managed 
care,  services  research,  and  community  planning.  When 
aggregated  across  individuals,  need  may  be  studied  on  at 
least  three  levels:  (1)  global  measures  of  need  and  unmet 
need,  such  as  whether  any  unmet  need  is  present  or  the 
number  of  items  with  unmet  need;  (2)  functional  domains, 
such  as  what  factors  are  associated  with  unmet  need  for 
services  designed  to  address  disruptive  behavior;  and  (3) 
individual  service  items,  such  as  factors  associated  with 
need  for  behavior  therapies.  Over  time,  the  INSA  will  be 
adapted  to  incorporate  new  knowledge  about  service 
strategies  and  effectiveness. 

References 

Bebbington,  P.  E.  (1990).  Population  surveys  of  psychiatric 
disorder  and  the  need  for  treatment.  Social  Psychiatry  and 
Psychiatric  Epidemiology,  25,  33-40. 

Brewin,  C.  R.  (1992).  Measuring  individual  needs  for  care 
and  services.  In  G.  Thronicroft,  C.  R.  Brewin,  & J.  Wing 
(Eds.),  Measuring  Mental  Health  Needs  (pp.  220-236). 
London:  Gaskell. 

Brewin,  C.  R.,  Wing,  J.  K.,  Mangen,  S.  P.,  Brugha,  T.  S.,  & 
MacCarthy,  B.  (1987).  Principles  and  practice  of  measuring 
needs  in  the  long-term  mentally  ill:  The  MRC  Needs  for 
Care  Assessment.  Psychological  Medicine,  17,  971-981. 

Burchard,  J.  D.  & Clark,  R.  T.  (1990).  The  role  of 
individualized  care  in  a service  delivery  system  for  children 
and  adolescents  with  severely  maladjusted  behavior.  Journal 
of  Mental  Health  Administration,  17,  48-60. 


439 


Bums,  B.  J.,  Angold,  A.,  & Costello,  E.  J.  (1992). 
Measuring  child,  adolescent,  and  family  service  use.  New 
Directions  for  Program  Evaluation,  54, 1 7-29. 

Duchnowski,  A.  J.  & Kutash,  K.  (1993).  Developing 
comprehensive  systems  for  troubled  youth:  Issues  in  mental 
health.  In  C.  M.  Nelson  & B.  Wolford  (Eds.),  Shakertown 
Symposiiom  II:  Developing  Comprehensive  Systems  for 
Troubled  Youth  . National  Coalition  for  Juvenile  Justice 
Services. 

Duchnowski,  A.  J.  & Friedman,  R.  M.  (1990).  Children's 
mental  health:  Challenges  for  the  Nineties.  Journal  of 
Mental  Health  Administration,  17(1),  3-12. 

Katz-Leavy,  J.,  Lourie,  I.  S.,  Stroul,  B.  A.,  & 

Zeigler-Dendy,  C.  (1992).  Individualizing  services  in  a 
system  of  care.  In  Profiles  of  local  systems  of  care  for 
children  and  adolescents  with  serious  emotional  disturbance 
(pp.  1-45).  Washington  DC:  CASSP  Technical  Assistance 
Center,  Georgetown  University. 

Klerman,  G.,  Olfson,  M.,  Leon,  A.  C.,  & Weissman,  M.  M. 
(1992).  Measuring  the  need  for  mental  health  care.  Health 
Affairs,  1 1(3),  23-33. 

Mangen,  S.  & Brewin,  C.  R.  (1991).  The  measurement  of 
need.  In  P.  E.  Bebbington  (Ed.),  Social  Psychiatry:  Theory, 
Methodology  and  Practice  (pp.  163-181).  New  Brunswick, 
NJ:  Transaction  Press. 

Pires,  S.  A.  (1990).  Sizing  Components  of  Care:  An 
Approach  to  Determining  the  Size  and  Cost  of  Service 
Components  In  a System  of  Care  for  Children  and 
Adolescents  with  Serious  Emotional  Disturbances. 
Washington  DC:  CASSP  Technical  Assistance  Center. 

Rivard,  J.  C.,  Perry,  N.,  & Hinkle,  K.  T.  (1994). 
Administrative  tools  for  operationalizing  a continuum  of 
care  for  children  and  youth  with  severe  emotional 
disturbance.  Journal  of  Mental  Health  Administration, 
21(3):240-252. 

Stroul,  B.  A.  & Friedman,  R.  M.  (1986).  A System  of  Care 
for  Severely  Emotionally  Disturbed  Children  and  Youth. 
Washington,  DC:  CASSP  Technical  Assistance  Center. 


Authors 

Rene  C.  Grosser,  Ph.D. 

Director,  Needs  Assessment  Unit 
NYS  Office  of  Mental  Health 
44  Holland  Avenue,  7th  Floor 
Albany  NY  12229 
Voice:  518/473-3861 
Fax:  518/473-3456 


440 


pbgross@iris.rfmh.org 

Alan  J.  Flisher,  M.Med.,  F.C.Psych.  (S.A.) 

Lecturer/Specialist 

Department  of  Psychiatry 

University  of  Cape  Town 

J5  Groote  Schuur  Hospital 

Observatory  7925 

Republic  of  South  Africa 

Voice:  011/27/21/404-2164 

Fax:  011/27/21/448-8158 

bh@ray.uct.ac.za 


Libertmi^!^KItasII^^^^^ne3mMin^^^^3s!^^'n^5^^^^t^^nnua^esearcl^onFerence 
Proceedings,  A System  of  Care  for  Children's  Mental  Health:  Expanding  the  Research  Base  (February 
26  - 28,  1996).  Tampa,  FL:  University  of  South  Florida,  Louis  de  la  Parte  Florida  Mental  Health 
Institute,  Research  & Training  Center  for  Children's  Mental  Health. 


441 


The  9tli  Annual  Research  Conference  Proceedings,  A System  of  Care  for  Children’s  Mental  Health:  Expanding  the 

Research  Base  (February  26  - 28,  1996). 


Assessing  Need  and  Planning  for  a System  of 
Care  in  a Culturally  Diverse  Urban  Community 


Authors 


Introduction  Method  & Results  Discussion  References 


Introduction 


There  is  a growing  urgency  to  develop  community-based 
systems  of  care  for  children  with  serious  emotional 
disturbances  as  the  nation  moves  to  provide  mental  health 
services  in  less  restrictive  and  less  costly  living 
environments  than  those  provided  in  institutional  settings. 

In  many  communities  across  the  country,  ethnic  minority 
children  represent  a significant  proportion  of  the  child 
population.  The  challenge  is  to  find  ways  of  incorporating 
these  children  into  service  delivery  systems  that  are  more 
culturally  sensitive  to  them. 

In  response  to  a call  for  a framework  that  would  guide  the 
development  of  community-based  systems  of  care,  Stroul 
and  Friedman  (1986)  proposed  that  "emotionally  disturbed 
children  should  have  access  to  a comprehensive  array  of 
services"  (p.  1 8)  coordinated  across  human  service  agencies 
such  as  education,  social  service,  health,  and  juvenile 
justice.  Cross,  Bazron,  Dennis,  and  Isaacs  (1989)  added  that 
"when  the  system  of  care  functions  as  an  integrated  support 
network"  (p.  53),  it  supports  and  enhances  the  delivery  of 
culturally  competent  services.  In  an  integrated  support 
network,  the  chances  of  considering  the  needs  of  both  ethnic 
minority  and  ethnic  majority  children  are  likely  to  be  much 
higher  than  in  a fragmented  and  duplicative  system.  Cross  et 
al.  (1989)  further  suggested  that  those  making  plans  for 
culturally  competent  systems  of  care  need  to  be  aware  of  the 
resources,  available  inside  and  outside  communities,  which 
can  promote  change.  Some  communities  have  a wealth  of 
resources  that  allow  them  to  develop  home-grown, 
self-sustaining,  and  culturally  competent  systems  of  care, 
but  many  communities  must  rely  on  outside  resources  to 
sustain  and  enhance  their  own  culturally  competent  care 
systems. 

Study  1 of  this  summary  provides  an  assessment  of  need  for 
children's  mental  health  services  in  the  Mott  Haven 
community  of  the  South  Bronx.  Such  assessment  will  be 
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used  as  a baseline  of  the  need  for  a community-based 
system  of  care  currently  being  developed  through  the 
Families  Reaching  in  Ever  New  Directions  (FRIENDS) 
Initiative,  a five-year  project  funded  by  a grant  awarded  by 
the  federal  Center  for  Mental  Health  Services.  The 
assessment  will  seek  to  answer  the  following  questions: 

• What  is  a reasonable  estimate  of  need  for  children's 
mental  health  services? 

• What  is  the  current  utilization  of  children's  mental 
health  services? 

• What  is  a reasonable  estimate  of  the  cost  of  mental 
health  services  currently  being  utilized? 

Study  2 describes  the  cultural,  personal,  and  service 
utilization  characteristics  of  Mott  Haven  children  who  are 
recipients  of  existing  mental  health  services  through  a 
correlational  analysis.  Such  description  is  viewed  as 
important  to  planning  for  a system  of  care  in  a culturally 
diverse  community  that  is  67%  Hispanic,  31%  African 
American,  and  2%  Caucasian.  The  analysis  will  address  the 
question:  To  what  extent  are  the  cultural  and  personal 
characteristics  of  Mott  Haven  children  related  to  the 
children's  utilization  of  mental  health  services? 

Method  and  Results 

Study  1 

The  New  York  State  Office  of  Mental  Health  (NYSOMH, 
1992)  has  developed  a population-based  approach  to 
estimate  the  need  for  mental  health  services  for  children  in 
specific  communities  across  the  state.  This  methodology 
resulted  in  estimates  of  the  number  of  children  needing 
services  during  a typical  week  of  the  year  for  each  of  seven 
types  of  mental  health  programs  (acute  inpatient  and  crisis 
residential,  home-based  crisis  intervention,  intermediate 
inpatient,  residential,  school-based  clinical,  intensive  case 
management,  and  clinical).  Low,  average,  and  high  need  for 
capacity  rates  per  100,000  children  in  the  population  were 
generated  based  on  geographic  differences.  The  high  rate 
was  applied  to  the  child  population  of  Mott  Haven  because 
of  the  presence  of  the  many  environmental  risk  factors  that 
are  believed  to  exacerbate  the  mental  health  conditions  of 
the  children.  The  first  column  of  Table  1 shows  the  estimate 
of  needed  capacity  for  each  of  the  seven  types  of  mental 
health  programs.  As  indicated,  a system  of  care  in  Mott 
Haven  would  need  to  serve  at  least  467  children  on  a 
continuous  basis.  These  children  represent  1 .4%  of  the 
population  of  32,793  children  and  youth  residing  in  Mott 
Haven.  This  is  consistent  with  Friedman's  (1987)  estimate 
that  a public  system  of  care  should  have  a capacity  to  serve 
between  1 to  2 percent  of  children  at  any  given  time. 

Second,  the  above  estimate  was  compared  to  the  actual 
utilization  of  mental  health  services  by  Mott  Haven  children 
and  youth  during  a one-week  period  in  1993,  as  reported  in 
the  Patient  Characteristics  Survey  of  the  NYSOMH.  As 
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the  Patient  Characteristics  Survey  of  the  NYSOMH.  As 
seen  in  the  second  column  of  Table  1.  only  160  children  and 
youth,  or  34%  of  the  estimated  number  of  children  who 
need  mental  health  services,  utilized  those  services. 

Third,  Table  2 reports  cost  estimates  for  each  of  the  seven 
types  of  mental  health  services  per  unit  of  utilization,  for 
total  units  utilized,  and  for  an  entire  year.  The  figure  of 
$4,234,932  provides  an  estimate  of  the  total  annual  cost  of 
sustaining  a comprehensive  array  of  services  for  children 
and  youth  ages  21  and  younger  in  the  Mott  Haven 
community. 

Study  2 

Plans  to  develop  a culturally  competent  system  of  care  are 
enhanced  by  information  on  how  cultural  and  personal 
characteristics  of  children  with  mental  health  needs  relate  to 
the  children's  utilization  of  mental  health  services.  Such 
relationships  were  explored  among  a sample  of  143  Mott 
Haven  children  and  youth  ages  17  and  under  who  utilized 
mental  health  services  during  a one- week  period  in  1993. 
Measures  of  cultural  and  personal  characteristics  included 
Ethnicity  (African  American  = 1,  Hispanic  = 2),  Primary 
Language  (English  = 1,  Other  language  = 2),  Age  Group, 
defined  as  eight  age  categories  between  0 to  17,  and  Gender. 
Children's  service  utilization  was  measured  through 
Program  Type  (Outpatient  = 1,  Inpatient  = 2),  Clinic  Visits, 
with  a 0 to  2 range.  Day  Treatment  Visits,  with  a 0 to  5 
range,  and  Location  (inside  or  outside  Mott  Haven).  The 
measures  of  visits  to  clinic  and  day  treatment  programs 
were  selected  because  clinic  and  day  treatment  were  the 
services  to  which  Mott  Haven  children  had  the  greatest 
access. 

Table  3 shows  Pearson  correlations  between  the  measures  of 
cultural  and  personal  characteristics  and  the  measures  of 
children's  service  utilization.  Ethnicity  had  a relatively 
strong  negative  relationship  with  Location  indicating  that 
African  American  children  had  a tendency  to  receive 
services  outside  Mott  Haven,  while  Hispanic  children 
tended  to  be  served  inside  Mott  Haven.  Primary  Language 
had  a moderate  negative  relationship  with  Location.  Since 
Spanish  was  the  primary  language  for  many  of  the  children, 
the  finding  suggests  that  these  children  had  better  access  to 
mental  health  services  within  Mott  Haven  rather  than 
outside  Mott  Haven.  Ethnicity  and  Primary  Language  were 
weakly  and  positively  related  to  Clinic  Visits  suggesting  a 
tendency  for  Hispanics  and  persons  speaking  Spanish  and 
other  languages  to  visit  clinics  more  often  than  African 
Americans.  Finally,  there  was  a weak  negative  relationship 
between  Age  Group  and  Clinic  Visits  suggesting  that 
younger  children  are  more  likely  to  visit  clinics  than  older 
children.  Among  possible  explanation  is  that  older  children 
and  youth  may  view  visits  to  a clinic  as  potentially 
stigmatizing. 

Discussion 
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Discussion 


Study  1 in  this  paper  suggested  the  utility  of  applying  a 
population-based  approach  to  estimating  the  need  for  a 
comprehensive  array  of  mental  health  services  for  children 
with  mental  health  needs  in  an  urban  community  such  as 
Mott  Haven.  The  estimate  of  need  for  services  in  Mott 
Haven  was  compared  to  the  children's  actual  use  of  services 
showing  that  there  may  be  a large  gap  between  the  need  and 
the  use  of  services.  An  estimate  of  the  cost  of  providing 
mental  health  services  to  Mott  Haven  children  in  1993  was 
possible  given  the  availability  of  information  on  the 
children's  use  of  services.  The  long  term  benefit  of  having  a 
baseline  estimate  of  mental  health  service  costs  is  that  it  will 
make  it  possible  to  compare  it  with  an  estimate  of  costs  after 
a new  system  of  care  for  children  has  been  implemented 
through  the  FRIENDS  Initiative.  At  such  time,  it  will  also 
be  possible  to  determine  whether  a new  system  of  care 
reduces  or  increases  costs  of  serving  children  with  mental 
health  needs  in  an  urban  community. 

The  findings  in  Study  2 provide  some  support  for  using 
personal  characteristics  and  cultural  measures  to  better 
understand  patterns  of  service  utilization  in  culturally 
diverse  urban  communities.  It  was  particularly  interesting  to 
see  the  extent  to  which  children  from  African  American  and 
Hispanic  backgrounds  appeared  to  use  mental  health 
services  differentially  inside  and  outside  Mott  Haven.  Such 
finding  suggests  the  need  to  develop  a fuller  understanding 
of  the  cultural  factors  that  mediate  the  differential  use  of 
services  by  children  with  mental  health  needs. 

Estimating  service  needs,  determining  rates  of  service 
utilization,  estimating  service  costs,  and  understanding 
personal  cultural  factors  that  mediate  service  use  are  all 
important  components  in  planning  systems  of  care  for 
children  with  mental  health  needs  that  are  also  culturally 
competent.  This  study  serves  as  a starting  point  for 
describing  the  relationship  between  these  indices. 
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Outcomes  data  have  increasingly  become  a meter  by  which 
mental  health  services  are  evaluated.  This  is  particularly  the 
case  in  this  age  of  managed  care,  and  calls  for  accountability 
in  service  delivery  where  providers  are  asked  to  justify 
services  on  the  basis  of  client  outcomes.  Outcomes  data  also 
are  crucial  for  quality  assurance  and  continuous  quality 
improvement  efforts,  as  embodied  in  concepts  such  as 
outcomes  monitoring  and  outcomes  management.  However, 
while  the  movement  toward  outcome  monitoring  should  be 
applauded,  evaluation  efforts  involving  outcomes  should  not 
be  naively  undertaken.  Service  providers,  administrators, 
and  evaluators  should  seriously  consider  a number  of 
important  theoretical  and  research  issues  that  can  limit 
conclusions  made  from  outcomes  data  and  research  studies. 
A number  of  principles  should  be  considered  when  planning 
outcome  studies  and  especially  when  interpreting  research 
results. 


Principles  to  Consider  when  Planning  Outcomes 

Recognize  that  "outcome"  refers  to  multiple  rather  than 
a singular  domain. 

The  word  "outcome"  has  been  used  many  times  without 
outlining  what  is  actually  meant.  A vague  use  of  the  term 
seems  common.  Multiple  outcomes  are  generally 
acknowledged,  but  the  apparent  lack  of  specificity  when 
discussing  outcomes  suggests  a need  for  further 
conceptualization. 

Hoagwood,  Jensen,  Petti,  & Bums  (in  press)  developed  an 
excellent  conceptual  model  of  outcomes  that  attends  to 
multiple  levels  (e.g.,  individual,  family,  and  community) 
and  contexts  (e.g.,  home,  school,  and  peer  group).  Their 
model  consists  of  five  domains:  (a)  symptomatic/diagnostic 
(e.g.,  depression,  aggression,  and  diagnosis);  (b)  functional 
(e.g.,  child's  adaptation  to  various  environments);  (c) 
consumer  perspectives  (e.g.,  client  satisfaction  and  family's 
quality  of  life);  (d)  environmental  (e.g.,  changes  in  marital 
relationship  or  school  environment);  and  (e)  services  and 
systemic  (e.g.,  reductions  in  out-of-home  placements, 
hospitalization  rates,  and  length  of  stay). 

I would  add  one  additional  outcome 
domain&endash;protection  of  child  and  family  rights  and 
dignity.  This  domain  would  include  no  harm  occurring  to 
the  child  or  family  during  treatment,  maintenance  of 
confidentiality,  provision  of  effective  services,  and 
outcomes  associated  with  consumer  empowerment,  such  as 
opportunities  for  family  involvement  in  developing  agency 
policies  and  services  as  well  as  family  involvement  in 
treatment,  especially  a significant  "voting"  role  in  treatment 
decisions. 

Choose  outcome  domains  that  coincide  with  the 
program  theory. 

It  is  important  to  select  outcome  domains  that  match  the 
service  program  being  evaluated.  What  outcomes  would  you 
reasonably  expect  to  be  associated  with  an  adequate 
implementation  of  your  program  theory?  What  additional 
outcomes  would  your  program  like  to  meet? 

This  principle  became  particularly  clear  as  a result  of  some 
work  recently  done  by  the  author  to  define  quality  indicators 
with  an  agency  that  provides  services  to  children  and 
adolescents  who  have  had  significant  contacts  with  the 
juvenile  justice  system.  They  were  asked  to  list  crucial 
elements  associated  with  producing  positive  clinical 
outcomes.  The  first  question  asked  was  what  was  meant  by 
clinical  outcomes,  as  they  recently  had  problems  developing 
a shared  understanding  of  this  term  in  a meeting  with  a 
managed  care  company.  This  agency  recognized  that  they 
might  not  be  able  to  significantly  impact  clinical  symptoms 
in  their  programs,  but  hoped  to  have  an  impact  on 
functioning,  especially  societal,  school,  and  vocational 
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functioning,  especially  societal,  school,  and  vocational 
functioning.  They  also  were  very  concerned  with  creating  a 
safe  environment  (i.e.,  protecting  client  rights)  for  those  in 
their  program. 

Principles  to  Consider  when  Interpreting 
Outcome  Results 

The  source  of  information  should  be  considered  when 
reflecting  on  outcome  results. 

Constructionist  theories  suggest  that  each  individual  has 
their  own  unique  perspective  on  the  world.  Truth  is  truly  in 
the  eye  of  the  beholder.  In  the  case  of  collecting  outcomes 
data,  it  seems  important  to  consider  how  different  sources 
(e.g.,  parents,  clinicians,  children,  and  teachers)  may 
contribute  different  information.  Many  possibilities  exist 
that  might  explain  why  we  would  obtain  different  outcome 
results  from  different  sources,  including:  (a)  each  source  has 
a different  sample  of  a particular  child's  behavior  from 
which  to  make  outcome  judgments,  and  (b)  points  of 
reference  for  making  judgments  differ. 

In  regards  to  the  first  reason,  parents  may  be  good  reporters 
of  some  behaviors,  but  may  be  less  aware  of  certain  aspects 
of  their  child's  life.  For  example,  the  amount  of  information 
a parent  has  about  a child's  school  and  social  life  may  vary 
considerably.  The  level  of  parent  involvement  would  affect 
the  validity  of  parent  reports  in  these  areas.  Another  area  of 
interest  concerns  the  validity  of  reports  of  externalizing  and 
internalizing  behaviors.  Parents  are  generally  thought  to  be 
better  reporters  of  externalizing  behaviors,  because  they  are 
more  likely  to  come  to  their  attention.  Adolescents  are 
viewed  as  better  reporters  of  internalizing  behaviors. 
Teachers  may  provide  valid  reports  of  academic  and  social 
behavior,  but  fewer  valid  reports  for  other  behaviors. 

Service  providers  likely  have  very  limited  knowledge  of  the 
child's  behavior  outside  of  the  service  setting. 

In  addition  to  interacting  with  the  child  in  different  contexts, 
different  sources  of  information  likely  have  different  points 
of  reference  from  which  to  make  their  judgments.  In  other 
words,  the  sample  to  which  the  child  is  being  compared  for 
making  judgments  is  likely  to  differ.  Response  categories  on 
various  scales  (e.g.,  somewhat  or  sometimes  true,  a lot  or  a 
little,  more  than  average  or  less  than  average)  may  have 
different  meanings  to  different  judges.  For  instance,  teachers 
are  very  familiar  with  a wide-range  of  children  and  may 
have  a different  sense  of  what  "average"  or  "a  lot"  mean, 
compared  to  parents  and  clinicians. 

Recognize  that  clinical  outcomes  are  not  solely  the  result 
of  services. 

Multiple  factors  impact  the  mental  health  of  children.  These 
include  family  influences,  peer  influences,  cultural 
influences,  economic  influences,  and  strengths  and 
resiliency.  These  factors  must  be  considered  as  additional 
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resiliency.  These  factors  must  be  considered  as  additional 
influences  on  changes  in  symptoms  and  functioning  that 
occur  between  intake  and  exit  from  services.  In  a review  of 
research  on  the  impact  of  disasters  on  the  psychological 
health  of  adults  and  children  (Salzer  & Bickman,  in  press), 
one  consistent  finding  is  that  disasters  are  associated  with 
high  levels  of  psychological  distress  immediately  following 
the  traumatic  event.  However,  as  the  community  and 
families  recover,  usually  without  the  aid  of  mental  health 
interventions,  psychological  problems  remit.  One 
explanation  for  these  results  is  based  on  a central  tenet  of 
ecological  theories«&endash;natural  systems  work  to  return 
to  stasis  following  a disruption. 

Children  and  families  entering  services  do  so  at  a time  of 
great  upheaval  and  are  generally  found  to  be  improved  at 
the  end  of  treatment.  However,  the  role  of  services  in 
bringing  about  such  changes  often  fails  to  be  separated  from 
non-treatment  effects.  The  natural  resiliency  and  resources 
of  children  and  family  systems  likely  contribute  to 
improvements  over  time.  Children  and  families,  either 
consciously  or  unconsciously,  work  to  make  changes  that 
alleviate  stressors,  with  and  without  help  from  service 
providers.  Children,  parents,  and  other  family  members  also 
rely  on  their  natural  resources,  such  as  friends  and  other 
family  members,  in  addition  to  the  support  and  guidance 
provided  by  the  service  system.  The  fact  that  even 
non-treatment  control  groups  may  improve  over  time 
supports  the  notion  that  natural  recovery  processes  are 
important  factors  to  consider  when  making  causal 
statements  about  the  relationship  between  treatment  and 
improvement. 

Use  of  control  groups  would  be  one  obvious  approach  to 
take  in  order  to  distinguish  between  natural  recovery 
processes  and  treatment  effects.  The  development  of 
long-term  norms  on  various  measures  for  those  with  severe 
problems  who  have  not  received  services  might  be  another 
approach.  The  clinical  outcomes  for  one  child,  gathered  at 
6-months  post-intake,  could  be  compared  to  the  6-month 
norms  for  children  who  also  scored  high  on  the  same 
instrument  at  time  1 but  received  no  treatment. 

Adequate  statistical  power  and  appropriate  procedures 
are  necessary  when  analyzing  data. 

Statistical  conclusion  validity  (Bickman,  1990)  is  a 
significant  problem  with  current  research  on  outcomes 
associated  with  measures  of  mental  health.  Statistical 
conclusion  validity  refers  to  the  researchers'  ability  to  make 
valid  statistical  conclusions,  including  the  ability  to  detect 
an  effect  when  one  is  present  (i.e.,  power)  and  when  one  is 
not  present  (i.e..  Beta  or  type  II  error).  One  concern  is  that 
many  studies  lack  sufficient  statistical  power  to  detect  small 
or  medium  effect  sizes,  due  to  small  samples.  This  may,  in 
fact,  account  for  some  of  the  null  results  and  inconsistent 
results  reported  in  the  literature  for  various  interventions. 
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While  it  depends  on  the  effect  size  that  one  expects  to  be 
associated  with  the  intervention,  in  general,  researchers 
should  strive  for  large  sample  sizes  whenever  possible. 

While  lack  of  statistical  power  may  limit  the  number  of 
studies  finding  positive  effects  for  community  interventions, 
a great  number  of  studies  also  include  multiple,  unplanned 
statistical  analyses  that  inflate  the  familywise  error  rate  to 
such  an  extent  that  erroneous  conclusions  might  be  made 
suggesting  an  intervention  is  effective  when  it  may  not  be 
(i.e.,  alpha  or  type  I error).  For  example,  it  appears  common 
for  researchers  to  conduct  multiple  univariate  tests  without 
the  proper  corrections  for  inflated  alpha  levels. 

Conclusions  are  limited  by  the  outcome  domains  you 
choose. 

Results  from  one  outcome  domain  cannot  be  used  as  a 
proxy  for  other  domains.  This  is  a particularly  important 
concept  to  keep  in  mind  when  attempting  to  assess  the 
clinical  effectiveness  of  services,  without  obtaining  data  on 
clinical  symptoms  or  functioning.  For  example,  reductions 
in  service  utilization,  such  as  the  number  of  out-of-home 
placements  or  decreased  rates  of  hospitalization,  may  be 
reported  to  imply  that  a service  system  or  a particular 
service  like  case  management  are  clinically  effective. 
However,  service  utilization  outcomes  may  predominantly 
reflect  modifications  in  system  priorities  and  policies 
regarding  level  of  care  or  length  of  stay  decisions  rather 
than  actual  changes  in  the  child's  behavior  (e.g.,  one  of  the 
philosophies  associated  with  a system  of  care  places  an 
emphasis  on  keeping  children  in  their  homes,  or  at  least 
their  home  communities,  and  this  philosophical  change 
alone  may  account  for  decreases  in  out-of-home  placements 
and  hospitalization  rates  regardless  of  changes  in  the  child's 
behavior).  While  extremely  important,  service  utilization 
outcomes  are  of  little  benefit  in  evaluating  the  clinical 
effectiveness  of  a system  of  care. 

There  appears  to  be  another  current  trend  toward  reporting 
serv'ices  to  be  effective  based  on  results  from  consumer 
satisfaction  instruments.  Consumer  satisfaction  is  vitally 
important  to  how  services  should  be  evaluated,  but  the 
relationship  between  satisfaction  and  clinical  outcomes  is 
unclear  (Lambert,  Salzer,  Bickman,  1996).  The  best  way  to 
assess  clinical  outcomes  may  be  through  the  use  of 
measures  assessing  symptoms  and  functioning. 
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Introduction 

E.  Jane  Costello 


Recent  changes  in  regulations  governing  access  to  mental 
health  care  resources  for  children  have  included 
requirements  that  children  not  only  have  a psychiatric 
disorder  but  also  be  functionally  impaired.  This  has  aroused 
interest  in  the  concept  of  functional  impairment  or  adaptive 
functioning  (i.e.,  its  history,  measurement,  and  significance 
as  an  indicator  of  need  for  services).  In  this  symposium,  (a) 
Dr.  Horwitz  traces  the  history  of  the  concept,  (b)  Dr. 
Costello  describes  its  prevalence  and  correlates  in  the 
community,  and  (c)  Dr.  Angold  discusses  evidence  for 
impairment  as  a marker  of  service  need. 
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The  Origin,  Meaning  and  Use  of  Adaptive 
Functioning  in  Defining  Mental  Health 
Problems 

Sara  M.  Horwitz,  K.  Hoagwood,  S.  S.  Sparrow  & E.  Triche 

The  concept  of  adaptive  functioning  grew  out  of  the  study 
of  mental  retardation.  Concerns  with  decreased  intellectual 
functioning  became  important  because  of  increased  societal 
demands  brought  on  by  industrialization,  urbanization,  and 
universal  public  education.  Early  work  around 
quantification  of  intellectual  abilities  revealed  variations  in 
functioning  not  uniformly  related  to  intellectual  levels. 
These  differences  in  functioning  were  related,  however,  to 
an  individual's  ability  to  perform  age-appropriate  social 
roles  and  became  incorporated  into  the  definition  of  mental 
retardation.  For  those  with  severe  and  persistent  mental 
illness,  the  need  to  differentiate  individuals  with  respect  to 
prognosis  and  services  planning  led  to  an  interest  in 
adaptive  functioning  and  its  eventual  inclusion  in  the 
multiaxial  classification  system  of  the  Diagnostic  and 
Statistical  Manual  (DSM),  3rd  Edition.  Unfortunately, 
measurement  of  adaptive  functioning  within  the  definition 
of  mental  disorders  is  underdeveloped  and  has  been 
confined  largely  to  a single  global  measure  of  functional 
impairment. 

Service  Needs  and  Use  for  Serious  Emotional 
Disturbance:  A Community  Study 

E.  Jane  Costello 

Many  federal  and  state  agencies  and  insurance  companies 
are  adopting  the  federal  definition  of  Serious  Emotional 
Disturbance  (SED)  as  the  criterion  for  child 
psychopathology  of  sufficient  severity  to  require  services. 
SED  is  defined  as  a DSM-defmed  psychiatric  disorder 
resulting  in  functional  impairment  that  substantially 
interferes  with  or  limits  the  child's  role  or  functioning  in 
family,  school,  or  community  activities. 

Methods 

In  the  Great  Smoky  Mountains  Study,  we  examined  the 
prevalence  of  SED  in  a representative  population  sample  of 
children  age  9-13,  randomly  selected  from  a largely  rural 
area  of  the  southeastern  United  States.  Interviews  with 
1 ,015  parent  and  child  pairs  were  conducted  by  trained 
interviewers  using  (a)  the  Child  and  Adolescent  Psychiatric 
Assessment  (CAPA)  which  assesses  DSM-III-R  disorders 
and  functional  impairment,  and  (b)  the  Child  and 
Adolescent  Services  Assessment  (CASA)  which  records 
service  use  across  five  service  sectors  (i.e.,  specialty  mental 
health,  education,  primary  care,  juvenile  justice,  and  child 
welfare).  Functional  impairment  was  also  evaluated  using 
the  Children's  Global  Assessment  Scale  (C-GAS)  and  the 
North  Carolina  Child  and  Adolescent  Functional 
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North  Carolina  Child  and  Adolescent  Functional 
Assessment  Scale  (CAFAS).  Prevalence  estimates  were 
adjusted  for  design  effects. 

Results 

Of  the  20.3%  of  the  population  who  had  one  or  more 
DSM-III-R  disorders,  55%  (1 1 .1%  of  the  population)  met 
criteria  for  SED  according  to  one  or  more  of  the  three 
measures  of  impairment  used  (i.e.,  CAPA,  C-GAS,  or 
CAFAS).  Boys  and  girls  were  equally  likely  to  have  SED. 
Of  children  in  the  sample  with  conduct  disorder,  78.8%  had 
SED,  compared  with  73.4%  of  those  with  depressive 
disorders,  91.3%  of  those  with  oppositional  defiant 
disorders,  and  73.1%  of  those  with  ADHD.  Disorders  least 
likely  to  be  associated  with  functional  impairment  were 
anxiety  disorders  (45.8%),  tics,  and  functional  enuresis 
(32.6%  each). 

Children  with  SED  used  services  at  a higher  rate  than  those 
with  only  a DSM  diagnosis  or  only  functional  impairment. 
However,  the  majority  of  children  with  SED  had  received 
no  mental  health  services  from  any  service  sector  during  the 
three  months  preceding  the  interview.  Of  children  with 
SED,  19.2%  had  received  services  in  the  specialty  mental 
health  sector,  29.1%  in  the  school  system,  6%  from  primary 
care  providers,  5.3%  through  child  welfare  agencies,  and 
2%  through  juvenile  justice.  In  total,  only  37.7%  of  children 
with  SED  received  mental  health  care  from  any  agency, 
compared  with  6%  of  children  with  only  a DSM  diagnosis 
or  functional  impairment. 

Conclusions 

Use  of  the  SED  criterion  will  identify  around  one  child  in 
ten  in  the  population,  depending  on  the  criteria  used.  These 
youth  are  much  more  likely  than  children  with  less  severe 
symptoms  to  receive  mental  health  care;  however,  only 
about  one  in  three  is  currently  receiving  any  form  of  care. 
Thus,  adopting  SED  as  the  definition  of  need  for  care 
identifies  a large  gap  between  need  and  availability  of 
services. 

The  Validity  of  Serious  Emotional  Disturbance 
as  a Marker  for  Need  for  Services 

Adrian  Angold 

The  addition  of  functional  impairment  to  diagnosis  as  a 
criterion  for  eligibility  for  various  treatment  and 
reimbursement  programs  reflects  an  attempt  to  focus  scarce 
resources  on  those  in  greatest  need.  In  this  paper  we  explore 
some  empirical  evidence  testing  the  hypothesis  that  children 
with  SED  have  greater  service  needs  than  children  with  (a) 
functional  impairment  associated  with  psychiatric 
symptoms  but  no  specific  DSM  diagnosis,  (b)  children  with 
impaired  functioning  but  few  or  no  psychiatric  symptoms. 
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(c)  children  with  a DSM  disorder  but  no  functional 
impairment,  or  (d)  children  with  neither  psychopathology 
nor  impairment. 

Method 

The  Great  Smoky  Mountains  Study's  representative 
population  sample  of  youth  aged  9-13(N=l,015)  was 
used  for  these  analyses.  Psychiatric  symptoms  were 
assessed  by  trained  lay  interviewers  using  the  Child  and 
Adolescent  Psychiatric  Assessment  (CAPA),  and  service 
use  across  five  service  sectors  was  recorded  using  the  Child 
and  Adolescent  Services  Assessment  (CASA).  Service  need 
was  defined  in  the  following  ways:  (a)  use  of  specialty 
mental  health  services;  (b)  use  of  school  counselors  and 
school  psychologists  for  emotional  and  behavioral 
problems;  (c)  parental  report  of  economic,  social,  or 
psychological  burdens  associated  with  the  child's 
symptoms;  (d)  report  by  parent  or  child  that  the  child  had  a 
problem  in  one  or  more  symptom  areas;  and  (e)  report  by 
parent  or  child  that  help  was  needed  because  of  such  a 
problem. 

Results 

Children  with  SED  had  the  highest  rates  of  use  of  specialty 
mental  health  and  school  services,  the  highest  rates  of 
burden,  reports  of  problems,  and  need  for  help.  However, 
children  with  functional  impairment,  with  or  without  some 
psychiatric  symptoms,  were  more  likely  than  children  with  a 
DSM-III-R  diagnosis  but  no  functional  impairment  to  meet 
the  various  criteria  of  need  for  services.  Between  the  first 
and  second  years  of  the  study,  there  was  a considerable 
amount  of  movement  between  categories  for  severity  of 
disorder. 

Conclusions 

While  the  combination  of  a DSM  diagnosis  and  functional 
impairment  (SED)  identifies  children  with  high  levels  of 
need  for  services,  service  needs  are  also  high  in  youth  with 
less  severe  psychopathology  but  marked  functional 
impairment.  If  these  children  are  excluded  from  access  to 
care  because  agencies  adopt  SED  as  the  criterion  for  access 
to  care,  many  children  in  need  will  be  excluded. 
Furthermore,  it  will  exclude  from  services  children  with  less 
severe  symptoms  but  high  levels  of  functional  impairment 
who  are  likely  to  move  into  full-blown  SED  without 
treatment. 
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Introduction 

Consumers,  providers,  and  policymakers  all  have  called  for 
the  monitoring  of  quality  in  order  to  ensure  that  lower  costs 
do  not  mean  low-quality  care.  The  assessment  of  quality  is 
also  a significant  element  of  the  continuous  quality 
improvement  movement,  a movement  that  is  crucial  for 
improving  mental  health  services. 


I 


Three  circumstances  inhibit  meaningful  monitoring  of 
quality  of  care  in  children's  mental  health.  First,  the 
indicators  compiled  to  date  do  not  extensively  cover  all 
quality  domains.  Second,  research  on  quality  of  care  in 
children's  health  services  is  conspicuously  lacking 
(McGlynn,  Halfon,  & Leibowitz,  1995).  Third,  the  concept 
of  quality  of  care  is  underdeveloped  to  the  extent  that 
current  approaches  to  monitoring  quality  may  fail  to  assess 
the  "goodness"  of  delivered  services.  This  summary  focuses 
on  the  last  concern. 
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The  Holy  Trinity  of  Structure,  Process,  and 
Outcome 

Most  approaches  to  evaluating  quality  are  based  to  some 
extent  on  Donabedian's  tripartite  model  of  quality.  This 
model  addresses  three  domains:  structure,  process,  and 
outcome  (Donabedian,  1980).  Structure  refers  to  "the 
relatively  stable  characteristics  of  the  providers  of  care,  of 
the  tools  and  resources  they  have  at  their  disposal,  and  of 
the  physical  and  organizational  settings  in  which  they  work" 
(Donabedian,  1980,  p.  81).  Five  structural  categories  are 
thought  to  be  important  in  assessing  the  quality  of  service 
structure:  (a)  Access;  (b)  Institutional  Characteristics;  (c) 
Provider  Characteristics;  (d)  Community  Characteristics; 
and  (e)  Client  Characteristics. 

Process  focuses  primarily  on  treatment  process,  including 
interpersonal  process  factors  and  technical  skill  in  the 
delivery  of  services.  Interpersonal  process  refers  to  the 
therapeutic  relationship  and  rapport,  communication, 
information  dissemination,  and  shared  decision-making  that 
occur  as  part  of  treatment.  Technical  skill  encompasses 
knowledge  of  state-of-the-art  intervention  techniques,  the 
ability  to  assess  which  intervention  provides  the  best  match 
for  the  client's  problems  or  diagnosis,  and  the  skill  to 
effectively  deliver  the  best  matching  intervention. 

While  Donabedian  and  other  quality-of-care  theorists  focus 
on  technical  and  interpersonal  factors  in  understanding 
process,  additional  quality  process  categories  should  also  be 
considered.  Stages-of-treatment-process  indicators  concern 
the  quality  of  intake  and  assessment,  treatment  planning, 
service  utilization  (e.g.,  # of  sessions/service  units,  length  of 
treatment,  and  service  patterns),  and  appropriate 
termination.  Service  process  can  also  refer  to  barriers  to 
treatment,  timeliness  of  treatment,  accountability  to 
consumers,  consumer  advocacy,  and  protection  of  client 
rights. 

The  last  component  of  quality  is  outcome.  Donabedian 
(1980)  defined  outcome  as  "a  change  in  the  patient's  current 
and  future  health  status  (symptoms  and  functioning)  that  can 
be  attributed  to  antecedent  health  care"  (p.  82).  Donabedian 
also  included  patient  attitudes  about  treatment,  such  as 
patient  satisfaction,  health-related  knowledge,  and 
behavioral  change  in  areas  that  contribute  to  health 
problems.  Protection  of  child  and  family  rights  and  safety 
can  also  be  added  as  another  important  outcome  category. 
This  would  include  the  maintenance  of  confidentiality,  the 
utilization  of  least  restrictive  services,  and  the  right  to 
receive  services  known  to  be  effective. 

Approach  to  MonitoringQuality  of  Care 

One  basic  concept  appears  to  underlie  all  approaches  to 
monitoring  quality  of  care:  that  quality  can  be  examined  by 
assessing  any  one  of  its  three  components.  This  concept  is 


461 


assessing  any  one  of  its  three  components.  This  concept  is 
consistent  with  Donabedian  (1980),  who  considered  the 
evaluation  of  structure,  process,  or  outcome  as  all  providing 
a similar  picture  of  quality  of  care.  Three  approaches  have 
been  taken  to  assess  quality  of  care:  the  use  of  standards  and 
guidelines,  peer  review,  and  outcomes  monitoring. 

The  evaluation  of  structure  and  process  indicators  has  been 
based  on  standards,  practice  guidelines,  and  report  cards 
developed  through  literature  reviews,  general  consensus, 
and  expert  panels.  A growing  number  of  organizations  and 
agencies  have  developed  or  are  in  the  process  of  developing 
standards,  guidelines,  and  report  cards:  JCAHO,  NCQA, 
AHCPR,  CHAMPUS,  SAIC,  CMHS,  AMBHA,  and  NAMI. 

Peer  review  is  a second  approach  used  to  assess  quality  of 
care  process,  and  more  specifically,  the  assessment  of 
appropriateness  of  care.  In  a review  of  the  literature,  Lee 
Sechrest  noted  that  peer  review  of  mental  health  care 
process  is  common  (Sechrest,  1987).  For  example, 
standardized  peer  review  procedures  have  been  used  to 
assess  appropriateness  of  psychiatric  hospitalization 
(Strumwasser  et  al.,  1991)  and  quality  of  outpatient  services 
(e.g.,  Daniels,  Kramer,  & Mahesh,  1995;  Hargrave  & Hiatt, 
1995).  Moreover,  researchers  affiliated  with  the  RAND 
Corporation  have  used  a "tracer  methodology"  involving  the 
clinical  review  of  medical  records  to  assess  quality  of  care 
(Wells  etal.,  1993). 

Finally,  outcomes  monitoring  has  grown  in  popularity  over 
the  past  few  years  as  a way  to  assess  quality  (Guadagnoli  & 
McNeil,  1994).  This  method  assumes  that  quality  care  has 
been  delivered  if,  for  example,  clients  are  found  to  have 
decreased  symptoms  and/or  increased  functioning  after 
treatment,  or  clients  report  they  are  satisfied  with  treatment. 

Problems  with  Current  Approaches  to  Monitoring 
Quality  of  Care 

Three  glaring  problems  are  evident  with  these  approaches  to 
assessing  quality  of  care.  First,  the  validity  of  structure  and 
process  indicators  remains  in  question,  because  the  extent  to 
which  these  indicators  are  linked  to  outcome  is  unknown. 
The  Institute  of  Medicine's  (lOM;  1991)  definition  of 
quality  care  referred  to  "the  degree  to  which  services  for 
individuals  and  populations  increase  the  likelihood  of 
desired  health  outcomes  and  are  consistent  with  current 
professional  knowledge"  (p.l).  Only  those  quality  indicators 
for  which  links  among  structure,  process,  and  outcome  exist 
can  be  referred  to  as  valid  (AHCPR,  1995;  Wells  & Brook, 
1989). 

Unfortunately,  structure  and  process  indicators  are  mostly 
based  on  expert  judgment,  which  may  reflect  what  we  call 
aesthetic  quality  and  may  not  be  associated  with  improved 
outcomes.  Despite  their  intuitive  attractiveness,  these 
indicators  may  have  no  functional  use.  For  example,  the 
setting  of  a program,  such  as  a spacious  and  pleasant 
physical  environment,  might  not  be  tied  to  a clinical 
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physical  environment,  might  not  be  tied  to  a clinical 
outcome  as  one  might  theorize.  Credentials  and  provider 
experience  might  be  another  example  of  aesthetic  quality. 
Neither  of  these  have  been  clearly  linked  to  better  clinical 
outcomes  (Berman  & Norton,  1985;  Durlak,  1979;  Stein  & 
Lambert,  1984;  Weisz,  Weiss,  Alicke,  & Klotz,  1987). 

Some  promising  links  between  service  processes  and 
clinical  outcome  have  been  found  for  outpatient 
psychotherapy.  However,  most  of  this  work  resulted  from 
laboratory  research.  Little  research  has  been  conducted  in 
community-based  settings.  It  is  our  concern  that  the  lack  of 
valid  indicators  may  result  in  meaningless  quality  assurance 
and  continuous  quality  improvement  efforts. 

A second  problem  with  the  current  assessment  of  quality 
concerns  peer  reviews.  Peer  review  approaches  to  assessing 
quality  have  been  criticized  for  being  unreliable.  For 
example,  studies  have  found  low  inter-rater  reliability  for 
peer  reviews  of  the  quality  of  outpatient  psychotherapy 
(Cohen  & Nelson,  1982;  Dali  & Clairbom,  1982).  Based  on 
his  review  of  the  research,  Sechrest  (1987)  concluded  that 
peer  review  is  of  limited  use  for  assessing  quality.  This  is 
due  to  the  fact  that  peer  review  is  based  on  intuition  and 
theoretical  biases  that  may  result  in  erroneous  conclusions. 
The  unreliability  of  panel  reviews  is  evident  in  studies  that 
examine  factors  impacting  the  evaluation  of 
appropriateness.  Research  has  shown,  for  example,  that 
panel  composition  (Fraser,  Pilpel,  Kosecoff,  & Brook, 

1994)  and  knowledge  of  clinical  outcome  before  judgments 
of  appropriateness  (Caplan,  Posner,  & Cheney,  1991)  have 
been  found  to  influence  peer  review  ratings  of 
appropriateness. 

A third  problem  with  current  quality  assessment  approaches 
is  that  outcome  monitoring  alone  does  not  provide  any 
information  about  the  quality  of  care  that  supposedly  led  to 
those  outcomes.  Interpreting  outcomes  should  be  done  with 
great  caution,  because  outcomes  may  have  occurred  as  a 
result  of  many  influences  other  than  the  services  provided. 
Threats  to  internal  validity  may  include  history  and 
maturation,  involvement  in  nonprofessional  programs,  or 
natural  recovery  processes.  Whereas  Donabedian  (1980) 
recognized  the  need  to  examine  outcomes  that  can  be 
attributed  to  antecedent  health  care,  mental  health  services 
researchers  often  do  not  attempt  to  distinguish  between 
services-related  outcomes  and  non-services-related  changes. 

Quality  as  Relationship  Between  Structure,  Process,  and 
Outcome 

In  order  to  refocus  our  attention  on  the  meaningful 
assessment  of  quality,  effort  needs  to  be  made  to  study  the 
validity  of  quality  indicators.  In  order  to  further  this 
research,  we  propose  a model  whereby  quality  is  viewed 
strictly  as  the  relationship  between  quality  indicators  (i.e., 
structure  and  process  variables)  and  quality  validators  (i.e., 
outcomes:  see  Figure  1).  This  is  in  contrast  to  viewing 
quality  as  existing  separately  within  each  component.  Given 
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quality  as  existing  separately  within  each  component.  Given 
this  conceptualization,  the  quality  of  a specific  structure  or 
process  variable  is  determined  by  the  extent  to  which  it  is 
related  to  some  outcome.  Given  this  understanding  of 
quality,  quality  of  care  research  would  examine  the  links 
between  quality  indicators  and  validators.  For  example, 
coordination  of  services,  a structural  variable,  may  be  found 
to  be  significantly  associated  with  decreased  symptoms  and 
increased  functioning.  Coordination  of  services  would  then 
be  viewed  as  a valid  quality  indicator  of  decreased 
symptoms  and  increased  functioning.  Interpersonal  process 
between  the  clinician  and  child/family  may  be  found  to  be 
significantly  associated  with  satisfaction,  but  not  the  other 
outcome  domains.  Therefore,  interpersonal  process  would 
be  viewed  as  a quality  indicator  of  satisfaction. 
Completeness  of  paperwork,  another  process  indicator,  may 
be  found  to  be  unrelated  to  any  indicator  of  outcome  and 
would  therefore  not  be  considered  a valid  quality  indicator 
of  clinical  outcome. 

Advantages  of  Adopting  this  Conceptualization  of 
Quality 

Adoption  of  this  conceptual  framework  of  quality  will 
benefit  the  field  in  a number  of  ways.  First,  and  most 
important,  measuring  valid  quality  indicators  and  making 
changes  in  response  to  this  data  (e.g.,  through  continuous 
quality  improvement  efforts)  will  mean  better  services  for 
children  and  families.  Second,  rather  than  making  broad 
statements  about  the  quality  of  a particular  service  or 
program,  providers  and  evaluators  will  be  able  to  explicitly 
state  what  outcomes  are  associated  with  specific  structural 
or  process  features.  For  example,  a particular  service  or 
program  will  be  able  to  say  that  they  have  short  waiting 
periods  that  have  been  shown  to  be  associated  with  high 
satisfaction,  offer  services  according  to  practice  guidelines 
that  have  been  found  to  be  associated  with  better  clinical 
outcomes,  and  follow  established  procedures  to  protect 
client  rights  to  confidentiality.  Third,  distinguishing 
between  outcomes  associated  with  the  various  indicators 
will  assist  policymakers  and  administrators  in  making  more 
informed  decisions  about  the  utility  of  certain  program 
features  over  others.  For  example,  improvement  efforts 
aimed  at  altering  process  indicators  associated  with 
decreased  symptoms  and  increased  functioning  might  be 
encouraged  over  those  indicators  associated  with  higher 
satisfaction. 

In  conclusion,  current  quality  assurance  and  continuous 
quality  improvement  efforts  are  limited  in  their  impact  to 
the  extent  that  the  quality  indicators  used  are  valid.  The  . 
validity  of  quality  indicators  is  crucial  if  we  are  to 
meaningfully  examine  quality  of  care. 
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Introduction 

The  Child  and  Adolescent  Functional  Assessment  Scale 
(CAFAS;  Hodges,  1990)  provides  information  on 
impairment  which  can  be  used  to  shape  the  course  of 
clinical  care  and  policy  decision-making.  It  provides  for 
child's  overall  functioning  score  as  well  as  scale  scores  for 
eight  psychosocial  areas  (i.e.,  school/work,  home, 
community  relationships,  moods,  self-harmful  behavior, 
substance  use,  and  abnormal  thinking).  The  CAFAS  is 
currently  being  used  to  describe  the  service  needs  of  youth 
in  care  and  to  assess  outcomes.  In  Tennessee,  an  innovative 
program  sponsored  by  the  Tennessee  Commission  on 
Children  and  Youth  uses  the  CAFAS  to  asses  a sample  of 
all  youth  in  the  state's  custody,  served  by  protective 
services,  mental  health,  juvenile  justice,  or  educational 
programs.  The  first  stage  of  an  outcome  study  being 
conducted  on  youth  served  by  the  Department  of  Mental 
Health  in  Missouri  is  also  presented. 
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CAFAS:  Evaluating  Statewide  Service 

Craig  Anne  Heflinger  & Celeste  G.  Simpkins 

The  Child  and  Adolescent  Functional  Assessment  Scale 
(CAFAS:  Hodges,  1990)  was  used  to  measure  psychosocial 
functioning  for  children  and  youth  in  state  care  as  part  of  the 
Children's  Plan  Outcome  Review  Team  (C-PORT).  The 
C-PORT  is  conducted  by  the  Tennessee  Commission  on 
Children  and  Youth  (TCCY)  as  an  evaluation  of  the  service 
system  for  children  in  custody  of  the  State  of  Tennessee. 

The  C-PORT  was  initiated  in  1994.  This  report  summarizes 
the  findings  of  the  1 994  and  1 995  data  collection  efforts  of 
child  psycho-social  functioning  that  included  846  children 
and  youth  ages  3 to  21  years  of  age.  For  the  overall 
C-PORT  findings,  see  TCCY  (1996). 

Method 

Sample 

The  C-PORT  uses  a proportional  stratified  sample  design. 
There  is  a base  population  of  approximately  12,000  children 
in  custody  at  any  given  time  being  served  by  any  of  14 
Assessment  and  Care  Coordination  Teams  (ACCT).  Within 
each  ACCT  region,  cases  are  assigned  to  one  of  27 
categories  based  on  age  (birth  - 5,  6 - 12,  and  13  or  more 
years),  current  placement  type  (family  home,  foster  home, 
or  group  placement),  and  adjudication  type 
(dependent/neglected,  unruly,  or  delinquent).  Cases  were 
randomly  selected  from  each  of  the  27  categories,  based  on 
the  proportion  within  the  ACCT  region  population. 

The  resulting  sample  for  1994  was  357,  and  for  1995,  it  was 
674,  for  a total  of  1 ,03 1 children  and  adolescents  (see  Table 
i).  The  CAFAS  was  completed  for  283  children  in  1994  and 
563  in  1995.  The  characteristics  of  the  CAFAS  sample  for 
1994-1995  are  shown  in  Table  1.  The  majority  of  these 
youth  were  13  years  old  or  over  (68%),  white  (69%),  male 
(57%),  dependent/neglected  (61%),  and  lived  in  a foster 
home  (38%)  or  group  residence  (39%). 

Source  of  Data 

Data  were  collected  by  C-PORT  staff  from  record  reviews 
as  well  as  interviews  with  the  child,  primary  caretaker, 
parent  (if  different  from  the  primary  caretaker),  and  teacher 
(if  appropriate)  from  a sample  of  children  in  custody  of  the 
state  of  Tennessee. 

Measurement  Instrument 

The  CAFAS  (Hodges,  1990)  is  an  interviewer-rated 
instrument  used  to  measure  children's  functioning  across 
five  domains:  role  performance,  thinking,  behavior  towards 
self  and  others,  moods  and  emotions,  and  substance  use. 
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self  and  others,  moods  and  emotions,  and  substance  use. 

I Two  additional  subscales  describe  the  current  caregiver's 
ability  to  provide  for  the  youth's  material  needs  and  for  his 
or  her  emotional/social  needs.  The  CAFAS  has 
demonstrated  good  interrater  reliability  (Hodges  & Wong, 
1996)  and  validity  (Hodges  & Gust,  1995;  Hodges  & Wong, 
in  press)  and  is  currently  being  used  by  numerous  states  to 
describe  children  receiving  state  funded  services  and  in  the 
Center  for  Mental  Health  Services  national  evaluation  of 
system  of  care  demonstration  projects. 

The  CAFAS  was  completed  at  the  end  of  a case  review  and 
interview,  based  on  information  from  the  structured 
interviews  and  case  reviews  of  the  C-PORT  protocol. 
Training  on  completion  of  the  CAFAS  was  conducted  by 
staff  of  the  Center  for  Mental  Health  Policy  at  Vanderbilt 
University.  Descriptive  statistics  were  used  to  develop  a 
profile  of  the  CAFAS  sample  and  functioning  levels  of  the 
youth  as  well  as  to  describe  the  1994-1995  differences. 

Results 

Measures  of  psychosocial  functioning  attempt  to  describe 
the  child's  ability  or  inability  to  function  in  his  or  her 
community  in  a variety  of  age-appropriate  ways.  Although 
many  of  the  children  were  rated  as  functioning  in  the 
average  range  for  their  age  for  specific  areas,  up  to  half  of 
them  demonstrated  some  type  of  impairment  in  each  of  the 
different  types  of  functioning  measured.  Two-thirds  (67%) 
were  rated  in  need  of  treatment,  in  contrast  to  the 
approximately  80%  of  children  who  were  given  a positive 
rating  of  emotional  well-being  as  part  of  the  C-PORT 
(TCCY,  1996).  That  rating  of  emotional  well-being, 
however,  does  not  attempt  to  address  the  actual  emotional 
health  or  problems  of  this  population  but  rather  whether  the 
service  system  has  identified  and  addressed  those  emotional 
needs  if  they  exist.  Therefore,  these  CAFAS  ratings  provide 
the  needed  descriptive  information  on  just  how,  and  how 
well,  these  children  and  youth  in  state  custody  are  actually 
functioning  on  a daily  basis. 

The  two  domains  in  which  the  children  exhibited  the  most 
problems  in  functioning  were  role  performance  (i.e.,  the 
effectiveness  with  which  the  child  fulfills  the  roles  most 
relevant  to  his  or  her  place  in  the  community)  and  behavior 
toward  self  or  others.  Two  thirds  of  the  children  (66%)  were 
rated  as  impaired  in  at  least  one  of  the  five  areas,  with  half 
receiving  impaired  ratings  in  two  or  more  areas.  Overall,  the 
CAFAS  scores  indicated  the  following  treatment  needs  for 
the  population  of  children  and  youth  in  state  care:  (a)  33% 
in  need  of  supportive  intervention;  (b)  30%  in  need  of 
short-term  treatment  (likely  on  an  outpatient  basis);  (c)  25% 
in  need  of  more  intensive  services  likely  in  excess  of  six 
months;  and  (d)  12%  in  need  of  intensive  services  and  likely 
needing  some  type  of  services  on  a longer  term  basis. 

Over  half  (53%)  of  the  children  had  a formal  mental  health 
diagnosis  reported,  and  of  these,  many  were  also  rated  with 
moderate  or  severe  impairment  in  psychosocial  functioning. 
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moderate  or  severe  impairment  in  psychosocial  functioning. 
Using  the  twofold  definition  of  the  Tennessee  Department 
of  Mental  Health  and  Mental  Retardation  that  requires  both 
of  these  documented  problems  to  be  classified  as  seriously 
emotionally  disturbed  (DMHMR  Priority  2),  29%  of  the 
children  could  be  so  classified. 

The  proportion  of  children  and  youth,  categorized  by 
custodial  department,  with  moderate  or  severe  impairment 
indicating  need  for  more  intensive  or  longer  term  mental 
health  services  were:  Department  of  Mental  Health  and 
Mental  Retardation  (83%),  Department  of  Youth 
Development  (58%),  Tennessee  Preparatory  School  of  the 
Department  of  Education  (41%),  and  Department  of  Human 
Services  (32%). 

The  relationship  between  child  age  and  psychosocial 
functioning  was  found  to  be  significant  with  younger 
children  (less  than  6 years  of  age)  demonstrating  less 
impairment.  In  both  of  the  two  older  age  groups,  6-12 
years  and  13  or  more  years,  substantial  levels  of  impairment 
were  reported.  Significant  differences  were  also  found  for 
gender  (i.e.,  males  were  reported  as  having  more 
psychosocial  impairments).  Race  of  the  children  was  also 
examined,  but  no  statistically  significant  differences  were 
found  (see  Table  2V 

The  type  of  residence  in  which  the  child  or  youth  currently 
was  placed  was  also  found  to  relate  significantly  to  the  level 
of  impairment  of  the  children  and  youth  in  all  areas. 

Children  in  group  placements  scored  as  significantly  more 
impaired  (mean  CAFAS  43.1)  than  children  in  family  (mean 
28.5)  or  foster  (mean  20.2)  homes.  Also  related  to  the 
child's  placement,  the  child's  level  of  functioning 
impairment  was  found  to  be  significantly  higher  when  he  or 
she  had  experienced  a greater  number  of  placements  (see 
Table  3).  In  other  words,  the  children  who  had  the  most 
instability  of  placement  were  those  with  the  greatest 
problems  in  their  ability  to  function  on  a daily  basis.  The 
children  who  had  been  in  their  current  placement  for  the 
shortest  amount  of  time  were  those  with  the  greatest 
psychosocial  functioning  impairment.  These  were 
frequently  the  children  who  had  been  in  multiple  placements 
so  that  the  placement  at  the  time  of  the  C-PORT  interview 
was  shorter  than  the  children  who  were  functioning  better 
and  staying  longer  in  their  placements. 

In  addition,  the  relationships  between  CAFAS  ratings  of 
psychosocial  impairment  and  the  C-PORT  ratings  of  child 
and  family  status  and  system  performance  were  examined. 
The  predicted  relationships  between  better  (less  impaired) 
functioning  and  positive  emotional  well-being,  positive 
educational/vocational  progress,  and  progress  achievement 
was  found  to  be  significant.  In  contrast,  children  and  youth 
with  greater  impairment  in  psychosocial  functioning  were 
found  to  be  more  likely  to  receive  negative  status  ratings  in 
emotional  well-being  or  educational/vocational  progress  or 
inadequate  status  in  progress  achievement. 
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inadequate  status  in  progress  achievement. 

The  CAFAS  also  provides  a rating  of  general  caregiver 
functioning.  The  ability  of  the  current  caregiver  to  provide 
for  the  basic  needs  of  the  child  includes  provision  of  food, 
shelter,  clothing,  medical  care,  and  safety.  The  social 
resources  of  a family  or  the  current  caregiver  are  also 
critical  to  child  development.  Almost  all  (95%)  of  the 
current  caregivers  were  rated  as  providing  basic  material 
resources  and  meeting  the  physical  and  safety  needs  of 
children  in  state  custody,  and  82%  were  rated  as  providing 
necessary  emotional  and  social  support  to  meet  the  child's 
developmental  needs. 

Discussion 

Several  additional  comments  are  needed  regarding  the 
performance  of  the  CAFAS  in  the  C-PORT  reviews.  First,  it 
is  obvious  from  the  above  findings  that  the  inclusion  of  the 
CAFAS  provides  much  more  information  about  the 
functioning  of  the  children  and  youth  in  state  custody  than 
reliance  on  the  single  child  characteristic  of  emotional 
well-being  (positive  vs.  negative  status).  The  CAFAS 
provides  a relatively  cost-effective  mechanism  for  allowing 
in-depth  description  of  the  needs  of  children  in  state  care  in 
the  domains  of  their  daily  functioning  including:  (a)  role 
performance  at  home,  in  school,  and  in  the  community;  (b) 
clear  thinking  and  thought  processing;  (c)  behavior  towards 
self  and  others;  (d)  moods  and  emotions;  and  (e)  substance 
use.  Second,  the  CAFAS  results  reported  above  "make 
sense,"  which  is  important  when  adding  a new  data 
collection  method,  in  that  the  results  correspond  to  other 
evidence  from  family  members  and  caseworkers  who  have 
described  the  problems  and  needs  of  these  children.  Thus, 
the  validity  of  the  CAFAS  in  this  setting  is  supported. 
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The  Utilization  of  the  Child  and  Adolescent 
Functional  Assessment  Scale  for  Assessing 
Program  and  Clinical  Outcomes,  Mental  Health 
Policy,  and  Child  Outcomes  in  Missouri 

La  Vonne  Daniels  & Lisa  Clements 

Introduction 

This  summary  describes  the  initial  findings  of  a statewide 
study  to  assess  outcomes  for  children  and  families  served  by 
agencies  funded  by  the  Missouri  Department  of  Mental 
Health  (DMH).  The  study,  which  is  part  of  a broad  based 
study  encompassing  mental  health,  substance  abuse,  and 
developmental  disabilities  services,  was  funded  by  the  state 
legislature. 

The  aim  of  the  study  is  to  assess  outcomes  for  a 
representative  sample  of  children  who  receive  services  in 
four  treatment  modalities:  (a)  residential  facilities,  (b)  acute 
inpatient  facilities,  (c)  outpatient  and  day  treatment 
programs,  and  (d)  intensive  in-home  programs.  The  design 
is  longitudinal  and  will  measure  client  functioning  at  two 
successive  points  in  time,  6 months  apart.  The  study 
includes  both  new  admissions  and  a sample  of  active  clients 
who  are  at  various  points  in  the  treatment  process.  In 
addition,  discharge  data  will  also  be  gathered  for  children 
served  in  acute  care  inpatient  and  intensive  in-home 
services. 

Method 

Measures  include  instruments  to  assess  functional  and 
clinical  status  of  the  child,  as  well  as  measures  of  family 
empowerment  and  family  satisfaction.  Instruments  for  the 
study  are  the  Child  and  Adolescent  Functional  Assessment 
Scale  (CAFAS;  Hodges,  1990),  the  parent  version  of  the 
Child  Behavior  Checklist  (CBCL;  Achenbach  & Edelbrock, 
1991),  the  Children's  Global  Assessment  Scale  (G-GAS; 
Schaffer  et  al.,  1983),  the  Family  Empowerment  Scale 
(FES;  Koren,  DeChillo,  & Friesen,  1992),  and  the  Family 
Satisfaction  and  Needs  Questionnaire  (Daniels,  1995).  In 
addition,  the  Admission  Checklist  for  Children  and  Youth,  a 
DMH  form  for  classifying  children  as  seriously  emotionally 
disturbed  or  acutely  disturbed,  and  a form  for  capturing 
demographic  data  were  also  utilized  in  the  study. 

The  study  was  implemented  October  1,  1995.  During  the 
months  of  October  and  November,  data  were  collected  for 
all  new  admissions  to  the  system  statewide.  Data  were  also 
collected  for  a representative  sample  of  active  clients  in 
each  of  the  major  treatment  settings.  Agencies  participating 
in  the  study  include  3 1 outpatient  programs,  4 acute  care 
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in  the  study  include  3 1 outpatient  programs,  4 acute  care 
inpatient  programs,  1 1 intensive  in-home  programs,  and  9 
residential  treatment  programs.  Prior  to  the  implementation 
of  the  study,  sessions  were  held  at  3 locations  in  the  state  to 
train  agencies'  staff  on  the  assessment  instruments  and  to 
describe  procedures  for  implementation  of  the  study. 
Protocols,  with  instructions  and  all  the  data  collection  forms 
and  instruments,  were  prepared  by  Missouri  Institute  of 
Mental  Health  (MIMH)  and  forwarded  to  local  agencies. 
Protocols  for  children  included  in  the  active  client  sample 
were  pre-labeled  with  identifying  information  for  each 
child. 

Preliminary  Findings 

As  of  February  1,  1996,  data  had  been  received  for  458 
children,  representing  all  four  target  groups.  Preliminary 
analyses  have  been  prepared  for  two  groups,  children  who 
were  active  clients  in  residential  care  (N  = 56)  and  children 
who  were  new  admissions  for  outpatient  or  day  treatment 
services  (N  = 159). 

Characteristics  of  Children  in  Residential  Care 

Children  in  residential  care  for  whom  responses  were 
received  were  primarily  Caucasian  males  (52%)  between  13 
and  15  years  of  age  (48%).  The  majority  were  in  the  custody 
of  parents  (57%),  while  30%  were  in  the  custody  of  the 
Department  of  Social  Services.  The  primary  diagnosis  for 
20%  of  the  group  was  conduct  disorder,  with  oppositional 
defiant  disorder  and  attention  deficit  hyperactivity  disorder 
accounting  for  an  additional  32%  of  the  group. 

The  children  had  high  levels  of  multi-agency  involvement. 
Sixty-six  percent  were  involved  with  Child  Protective 
Services,  and  86%  were  special  education  students.  Only  a 
small  proportion  (two  percent)  were  reported  to  be  involved 
with  Substance  Abuse  Agencies. 

Characteristics  of  Children  in  Outpatient  or  Day  Treatment 
Care 

Children  who  were  admitted  for  outpatient  or  day  treatment 
care  during  the  data  collection  period  were  primarily 
Caucasian  males  (56%).  Sixty-six  percent  were  ages  10-18 
years  of  age.  The  majority  were  in  the  custody  of  parents 
(86%).  Thirty  percent  had  previously  been  a client  of  DMH. 

The  primary  diagnosis  for  26%  of  the  group  was  attention 
deficit  hyperactivity  disorder,  with  oppositional  defiant 
disorder  accounting  for  an  additional  20%  of  the  group.  The 
children  had  some  multi-agency  involvement;  23%  were 
involved  with  Child  Protective  Services,  and  30%  were 
special  education  students.  Similar  to  children  in  residential 
care,  only  two  percent  were  reported  to  be  involved  with 
Substance  Abuse  Agencies.  While  primary  living  setting  for 
children  three  months  prior  to  data  collection  was  with  their 
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family  (86%),  14%  also  had  psychiatric  inpatient  care 
within  the  same  time  period. 

Discussion 

At  this  point  in  the  study,  only  descriptive  analyses  of  the 
data  are  appropriate.  This  is  especially  true  since  the  return 
rate  for  the  outpatient  group  is  relatively  low,  and  group 
sizes  are  very  different.  However,  some  differences  between 
the  outpatient/day  treatment  admissions  group  and  the 
residential  active  client  group  can  be  noted. 

Children  in  residential  care  were  older  than  children 
admitted  for  outpatient/day  treatment  and  were 
predominantly  male.  The  residential  treatment  group  were 
much  more  likely  to  have  been  involved  with  Child 
Protective  Services  (66%  vs  23%)  and  much  more  likely  to 
be  receiving  special  education  services  (86%  vs.  30%). 
Children  in  the  residential  treatment  group  also  had 
substantially  higher  levels  of  multi-agency  involvement 
(75%  vs.  30.8%). 

Among  the  residential  active  client  group,  20%  of  the  group 
had  a primary  diagnosis  of  conduct  disorder,  whereas  only 
2%  of  children  in  the  outpatient/day  treatment  admission 
group  had  conduct  disorder  as  a primary  diagnosis.  In 
contrast,  a primary  diagnosis  of  attention  deficit 
hyperactivity  disorder  was  more  likely  among  the 
outpatient/day  treatment  group  (26%)  than  for  the 
residential  group  (16.1%). 

Results  of  the  CAFAS  revealed  differences  between  the 
residential  treatment  group  and  the  outpatient/day  treatment 
admissions  group  as  well.  Although  scores  for  Role 
Performance  indicated  that  the  majority  of  both  groups  fell 
within  the  severe  impairment  category,  82%  of  the 
residential  treatment  group  compared  to  59%  of  the 
outpatient/day  treatment  admissions  group  scored  at  the 
severe  level  of  impairment.  The  Role  Performance  subscales 
scores  indicate  a substantial  proportion  of  both  groups 
(37.5%  for  the  residential  group,  41%  for  the  outpatient/day 
treatment  group)  were  having  severe  problems  at  school  or 
work.  However,  a far  greater  proportion  of  the  residential 
group  scored  at  the  severe  impairment  level  on  the  Home 
subscale  (80%)  compared  to  the  outpatient  admission  group 
(47%).  Although  more  than  fifty  percent  of  the  children  in 
both  the  outpatient  and  residential  groups  scored  at  the  "no 
impairment"  level  for  the  Community  subscale,  the 
residential  group  had  a greater  percentage  of  children 
scoring  at  the  severe  impairment  level  for  this  subscale. 

Another  notable  difference  in  CAFAS  scores  for  the 
residential  and  outpatient  groups  was  on  the  Substance  Use 
Scale.  Children  in  the  outpatient  group  had  a greater 
proportion  of  the  group  scoring  at  the  severe,  moderate,  and 
mild  impairment  levels  for  the  Substance  Use  scale.  No 
children  in  the  residential  group  were  reported  to  have 
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severe  impairment  levels,  while  25%  of  the  outpatient  group 
scored  at  this  level.  Four  percent  of  the  residential  group 
scored  at  the  moderate  level  of  impairment,  while  45%  of 
the  outpatient  group  scored  at  this  level.  When  considered 
across  impairment  levels,  70%  of  the  outpatient  group 
scored  in  the  moderate  or  severe  impairment  level  for 
Substance  Use. 

The  data  presented  above  are  for  only  two  of  the  four  groups 
in  the  study.  The  next  step  is  to  analyze  data  for  the  acute 
inpatient  care  and  intensive  in-home  groups  and  to  analyze 
data  for  the  Family  Empowerment  Scale  and  the  Family 
Satisfaction  and  Needs  Questionnaire.  Also,  in  the 
immediate  future,  follow-up  will  be  initiated  to  assure  that  a 
statistically  representative  sample  is  achieved.  In  April  and 
May  of  1 996,  follow-up  data  will  be  collected  for  the  entire 
population  in  the  study. 
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Introduction 

In  1994,  Kaleidoscope,  one  of  the  preeminent  child  welfare 
agencies  in  the  nation,  received  a grant  from  the  Annie  E. 
Casey  Foundation  to  evaluate  its  Satellite  Family  Outreach 
Program.  This  program's  priority  is  to  provide  services  to 
children  and  families  presenting  many  challenges  to  their 
systems  of  care.  The  children  and  parents  have  often  been 
diagnosed  as  seriously  disabled,  disturbed,  or  incorrigible 
and  are  in  need  of  intensive  treatment  service  efforts  toward 
integrating  them  into  their  home  communities  and/or  family 
life.  The  services  Kaleidoscope  provides  are  based  on  the 
philosophy  of  Wraparound  Care.  The  key  principles  of  care 
are:  (a)  unconditional  care,  (b)  intensive  case  management, 
(c)  individual  planning,  (d)  family  involvement,  (e)  flexible 
funding,  and  (f)  cultural  competence. 


The  Satellite  Program  uses  these  principles  in  serving  their 
children  and  families.  A major  component  of  these 
principles  of  care  is  to  identify  and  build  upon  behavioral 
strengths  of  the  children  and  families.  In  developing  the 
evaluation  plan  and  choosing  appropriate  instruments  for 
Satellite,  the  evaluation  team  noticed  that  limited  numbers 
of  assessment  instruments  were  available  to  measure 
children's  emotional  and  behavioral  strengths.  Additionally, 
those  instruments  that  did  exist  lacked  acceptable  validity 
and  reliability. 


Many  of  the  behavior  assessment  scales  currently  used  in 
education  and  psychology  are  deficit  oriented.  While  these 
scales  are  helpful  for  identifying  children  in  need,  they  may 
not  be  helpful  in  developing  a child's  treatment  plan  or 
goals.  However,  strength-based  assessment  can  be  used  to 
identify  a child's  competencies  and  therefore  be  useful  in 
developing  his/her  treatment  plans  and  goals. 

Strength-based  assessment  views  children  and  families  as 
individuals  with  unique  talents,  life  events,  and  needs.  From 
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an  ecological  perspective,  strength-based  assessment 
evaluates  children  and  families  across  various  life  domains 
(e.g.,  family,  social,  school,  and  psychological). 

We  began  to  develop  the  Behavioral  and  Emotional 
Strengths  Scale  (BESS)  to  provide  professionals  with  a 
valid  and  reliable  strength-based  assessment  instrument. 

The  BESS  is  a 68-item  scale  that  identifies  children's 
behavioral  strengths  on  five  dimensions:  (1)  self-control,  (2) 
affective  development,  (3)  family  involvement,  (4)  school 
performance,  and  (5)  self-confidence.  The  scale  is  designed 
for  teachers  and  direct  service  providers  to  complete  on 
children  with  whom  they  work.  Each  item  is  comprised  of  a 
statement  describing  a strength  children  may  demonstrate. 
The  respondent  then  rates  the  child  on  a Likert-type  scale  in 
terms  of  how  well  the  item  describes  the  child  (i.e.,  0 = Not 
at  All  Like  the  Child  to  3 = Very  Much  Like  the  Child). 
Completion  time  is  approximately  10-15  minutes.  The  item 
scores  are  then  summed  to  determine  the  child's  total  score. 
To  date,  the  content  validity,  item  discrimination,  and  factor 
structure  of  the  BESS  have  been  established. 

Content  Validity 

The  content  validity  of  the  BESS  was  determined  in  several 
steps.  First,  in  order  to  focus  on  important  areas  of  strength 
and  include  items  meaningful  to  practitioners,  250 
professionals  from  mental  health  and  social  service  agencies 
nationwide  were  mailed  surveys  in  which  they  were  asked 
to  list  up  to  10  behaviors  that  children  and  adolescents 
engage  in  when  they  begin  to  show  improvement.  The 
survey  respondents  listed  over  1 ,200  behaviors.  The  original 
list  was  then  reduced  to  190  by  combining  similar  items. 
Next,  these  190  items  were  grouped  into  15  categories  of 
behavioral  strengths  (e.g.,  self  responsibility,  school 
performance,  peer  relations,  views  of  others).  Based  on 
these  categories,  lists  comprised  of  7-9  items  were 
constructed  for  use  in  further  surveys.  Using  the  lists  of  7-9 
items,  a second  national  mailing  was  made  to  an  additional 
400  mental  health  and  social  service  staff.  These 
professionals  were  asked  to  rank  order  the  190  statements 
(each  respondent  ranked  three  lists)  with  regard  to  how 
important  they  were  to  a child's  behavioral  strengths. 

Finally,  based  on  median  and  quartile  scores  computed  for 
each  item,  another  63  items  were  deleted  (i.e.,  those  with 
high  median  and  high  quartile  scores).  The  resulting  scale 
consisted  of  127  items  that  described  the  behavioral 
strengths  that  a child  may  demonstrate. 

Item  Discrimination 

In  the  spring  of  1995,  a study  was  conducted  to  establish  the 
ability  of  the  BESS  to  discriminate  between  children  with 
and  without  serious  emotional  disturbance  (SED).  As  part  of 
this  study,  teachers  and  direct  service  providers  were  asked 
to  complete  the  127-item  version  of  the  BESS  on  children 
with  SED  (N  = 148)  and  children  without  SED  (N  = 1 10). 
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with  SED  (N  = 148)  and  children  without  SED  (N  = 110). 


One-way  analyses  of  variance  (ANOVAs)  and  Pearson 
Chi-Square  analyses  were  conducted  on  each  item.  The 
ANOVAs  were  significant  for  each  item.  The  children  with 
SED  scored  significantly  lower  on  each  item  than  did  the 
children  without  SED.  To  perform  a chi-square  analysis,  the 
response  choices  on  the  Likert-scale  were  collapsed  for 
comparison.  The  "Not  at  All  Like"  choice  remained  a single 
category,  the  other  three  choices  (i.e.,  "Not  Much  Like", 
"Like",  and  "Very  Much  Like")  were  combined  into  a 
general  "Like"  category.  A 2 ("Not  at  All  Like"  v.  "Like")  X 
2 (SED  V.  Non-SED)  Pearson  Chi-Square  test  was 
conducted  for  each  item.  The  analysis  identified  37  items  on 
which  children  with  SED  and  those  without  SED  did  not 
differ.  An  additional  1 0 items  were  eliminated  because  a 
sufficient  number  of  cells  with  the  minimum  expected 
frequency  for  a specific  item  was  not  obtained.  These  items 
were  discarded  resulting  in  an  80  item  version  of  the  BESS. 

Factor  Structure 

Having  determined  the  content  validity  and  item 
discrimination,  we  conducted  factor  analyses  to  determine 
the  important  domains  of  behavioral  strengths  assessed  by 
the  BESS.  The  factor  analyses  occurred  in  two  steps.  First, 
using  the  data  set  from  the  258  children  who  had  scales 
completed,  an  oblique  factor  analysis  with  item  loadings  set 
at  .40  was  conducted.  Nine  factors  emerged.  Second,  further 
factor  analyses  were  conducted  with  eigenvalues  set  at  1.5, 
item  loadings  set  at  .40,  and  four  items  set  as  the  minimum 
number  of  items  in  any  factor.  Five  meaningful  factors 
comprised  of  68  items  were  identified:  (1)  Self-Control,  (2) 
Affective  Development,  (3)  Family  Involvement,  (4)  School 
Performance,  and  (5)  Self-Confidence.  See  Figure  1 for 
items  in  each  factor. 

Current  Efforts 

Currently,  efforts  are  being  made  to  establish  norms  for  the 
Behavioral  and  Emotional  Strengths  Scale  (BESS).  As  of 
May  1996,  the  scale  has  been  completed  on  approximately 
1 800  children  with  and  without  SED  nationwide. 
Additionally,  a number  of  studies  to  determine  the  validity 
and  reliability  of  the  BESS  have  been  completed.  Three 
validation  studies  have  been  completed.  For  each  study, 
teachers  or  direct  service  providers  completed  the  BESS  and 
either  (a)  the  Teacher  Report  Form  (N  = 84),  (b)  the  Harter 
Self-Perceptions  Profile  for  Children  (N  = 78),  or  (c)  the 
Walker-  McConnell  Scale  of  Social  Competence  and  School 
Adjustment  (N  = 71)  on  up  to  10  of  their  students.  Studies 
to  establish  the  interrater  and  test-retest  reliability  have  also 
been  conducted.  In  these  studies,  teachers  and  direct  service 
providers  completed  the  BESS  for  up  to  10  students 
common  to  both  respondents  and  at  two  different  intervals. 
Preliminary  analyses  are  currently  underway. 
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Introduction 

As  mental  health  treatment  programs  come  under  closer 

scrutiny  by  payers  who  have  a range  of  options  for 
therapeutic  services,  well-designed  evaluations  of  the 
clinical  utility,  acceptability,  and  societal'^'alue  of  services 
are  needed  (Pfeiffer  & Strzelecki,  1990).  This  paper 
describes  development  of  a module  to  monitor  the  outcomes 
of  treatment  of  adolescents  with  emotional  or  behavioral 
problems. 


Although  an  estimated  $5  billion  is  spent  each  year  in 
providing  mental  health  services  to  children  and  adolescents 
(Bums,  Taube,  & Taube,  1990;  Rice,  Kelman,  Miller,  & 
Dunmeyer,  1990),  there  is  virtually  no  evidence  that  routine 
mental  health  care  for  youth  is  effective.  While  some 
isolated  studies  offer  optimistic  conclusions  about  the 
benefits  of  some  forms  of  treatment  (Cause  et  al.,  1994; 
Clark  et  al.,  1994;  Henggeler,  Melton,  & Smith,  1992; 
Scherer,  Brondino,  Henggeler,  Melton,  & Hanley,  1 994), 
other  evaluations  suggest  that  intensive  treatment  programs 
show  minimal  benefits  compared  to  standard  care  (Bickman 
et  al.,  1995;  Heneghan,  Horwitz,  & Leventhal,  1995).  Most 
treatment  programs  which  are  considered  promising  have 
yet  to  be  evaluated,  partly  because  no  comprehensive, 
standardized  set  of  instmments  have  been  developed  for 
adolescents  that  are  brief,  easily  administered,  and 
inexpensive.  To  that  end,  the  Adolescent  Treatment 
Outcomes  Module  (ATOM)  has  been  constmcted.  The 
objective  of  the  ATOM  is  to  (a)  provide  reliable  and  valid 
information  about  outcomes  of  care,  (b)  be  applicable  across 
severity  levels,  clinical  settings,  interventions,  and 
population  groups,  and  (c)  be  brief  and  inexpensive  to 
administer. 

Outcomes  modules  are  sets  of  standardized,  validated 
instruments  designed  to  facilitate  the  routine  and  systematic 
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gathering  of  data  on  patient  response  to  treatment  (Smith, 
Rost,  Fischer,  Bumam,  8c  Bums,  in  press).  The  ATOM  is 
the  sixth  in  a series  of  outcomes  modules  developed  by  the 
NIMH  Center  for  Rural  Mental  Health  Services  Research, 
University  of  Arkansas  for  Medical  Sciences.  Steps  in  the 
development  of  the  module,  components  of  the  module,  and 
preliminary  data  on  a small  sample  are  presented. 

Steps  in  Module  Development 

Module  development  began  with  an  extensive  review  of 
diagnostic,  prognostic,  health  service,  and  methodologic 
issues  of  adolescent  treatment  outcome  (Robbins  & Taylor, 
1995).  A multi-disciplinary  panel  of  experts  was  convened 
to  advise  the  development  team  on  critical  components  of 
the  module.  Experts  were  recmited  from  child  psychiatric 
epidemiology,  functional  measurement  in  children  and 
adolescents,  child  mental  health  services  research, 
stmctured  psychiatric  diagnostic  instmments  for  children, 
and  child  psychotherapy  research.  Based  on  panel 
recommendations,  a draft  of  the  module  was  composed  and 
presented  to  panel  members  for  evaluation.  Recommended 
changes  were  made,  and  a pilot  study  to  gather  validating 
data  has  been  undertaken. 

Module  Components 

The  module  is  composed  of  seven  domains:  (a)  focal 
problems;  (b)  diagnostic  assessment  to  establish  caseness  in 
five  diagnostic  categories  (i.e.,  anxiety,  depression, 
oppositional  defiant  disorder,  conduct  disorder,  ADD);  (c) 
multidimensional  assessment  of  functional  impairment;  (d) 
family  burden;  (e)  acceptability  of  treatment;  (f)  prognostic 
or  risk-adjustment  factors  that  may  influence  treatment 
outcome;  and  (g)  assessment  of  the  amount  and  quality  of 
treatment  received.  These  domains,  listed  in  Table  1.  have 
been  identified  as  most  central  to  understanding  outcomes 
of  treatment  for  emotional  and  behavioral  problems 
(Hoagwood,  Jensen,  Petti,  & Bums,  1996). 

Focal  Problems 

Youth  are  referred  to  clinics  not  because  of  a particular 
diagnosis,  but  because  parents,  teachers,  or  others  are 
concerned  about  specific  problems  they  are  having  at  home, 
school,  or  elsewhere  (Weisz  & Weiss,  1989).  The  module 
identifies  the  primary  reasons  for  initiating  treatment,  the 
seriousness  of  these  problems,  and  tracks  change  in 
presenting  problems  following  treatment. 

Caseness  and  Symptom  Severity 

It  is  not  feasible  within  the  scope  of  the  module  to  gather 
sufficient  data  to  make  diagnoses  of  disorders  using  full 
DSM-IV  criteria.  Therefore,  results  of  item  analyses  on  four 
large  data  sets  were  used  to  generate  a much  reduced  set  of 
symptoms  that  are  used  to  identify  likely  cases  of 
generalized  anxiety,  separation  anxiety,  major  depression. 
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generalized  anxiety,  separation  anxiety,  major  depression, 
oppositional  defiant  disorder,  conduct  disorder,  and 
attention  deficit  hyperactivity.  These  symptom  items  have 
been  supplemented  with  representative  symptoms  of  each 
disorder,  selected  on  the  basis  of  prevalence,  to  produce  a 
measure  of  symptom  severity. 

Functional  Impairment 

Therapeutic  outcomes  in  adolescent  mental  health  include 
both  resolution  of  symptoms  and  enhanced  functioning. 

Items  designed  to  address  domains  of  functioning  were 
drawn  from  multiple  sources  including  the  Brief  Impairment  j 
Scale  designed  by  Hector  Bird  (Bird,  1995)  from  the 
tradition  of  the  Columbia  Impairment  Scale  (CIS;  Bird, 
Shaffer,  Fisher,  Gould,  & Staghezza,  1993),  and  the  Child 
Health  Questionnaire  (CHQ:  Kurtin,  Landgraf,  & Abetz, 
1994).  The  ATOM  also  monitors  sentinel  indicators, 
relatively  rare  negative  events  caused  directly  or  indirectly 
by  emotional  or  behavioral  problems.  Sentinel  indicators, 
adapted  from  draft  items  of  the  UNOCCAP  study,  include 
inability  to  remain  in  the  home,  expulsion  or  suspension 
from  school,  and  frequent  arrests.  The  following  domains  of  I 
functioning  are  assessed:  j 

• Functioning  in  the  family.  In-home  placement  is  the 
goal  of  many  residential,  therapeutic  foster  care  and 
family  preservation  programs  and  is  the  primary 
outcome  measure  of  many  evaluations  (Gabel  & 
Shindledecker,  1992).  Ability  of  the  adolescent  to 
remain  in  the  home,  relationships  with  family 
members,  responsibility  at  home,  and  ability  to  follow  ; 
home  safety  rules  are  measured. 

• Functioning  in  school.  School-related  outcomes 

j include  acceptance  of  teacher  authority,  academic 

i progress,  and  completion  of  seatwork.  Parental  j 

involvement  through  frequent  calls  to  come  to  school  i 
! for  behavioral  conferences  or  to  pick  up  an  unruly 

: child  is  also  included,  as  are  sentinel  indicators  of 

I expulsion  and  suspension. 

j • Community  functioning.  Successful  functioning  in 
I the  community  is  measured  on  a sentinel  level  by 

I ability  to  comply  with  societal  laws.  Number  of  | 

I contacts  with  the  criminal  justice  system,  arrests,  time  ! 

I till  arrest,  incarceration,  and  residential  placement  are 

! determined.  Additionally,  measures  of  involvement  in  | 

I community  activities  and  use  of  leisure  time  are  } 

included. 

• Functioning  with  friends.  Peer  quality  is  assessed  by  | 

determining  the  ability  of  the  teen  to  make  friends 
who  do  not  normally  break  rules  and  laws  and  are  not 
regularly  in  trouble  with  authority  figures.  Peer  | 

relations  are  addressed  by  asking  both  parent  and  the 
adolescent  whether  (s)he  is  able  to  make  friends  and 
get  along  with  the  friends  (s)he  has  made. 

I 

Family  Burden 
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Burden  of  the  family  in  dealing  with  the  psychopathology  of 
an  adolescent  is  measured  by  the  Burden  Assessment  Scale 
developed  by  Reinhard  and  colleagues  (Reinhard,  Gubman, 
Horwitz,  & Minsky,  1994). 

Satisfaction  with  Care 

Satisfaction  with  mental  health  care  services  is  measured  by  i 
the  Client  Satisfaction  Questionnaire  (CSQ)  modified  by 
DeChillo  for  children  (Larsen,  Attkisson,  Hargreaves,  & 

Tuan,  1979).  The  CSQ  is  a brief  well-validated  instrument 
for  use  with  parents. 

1 

Prognostic  or  Risk-adjustment  Factors 

Factors  associated  with  successful  outcomes  of  care  were 
identified  from  the  literature  and  from  consultation  with  the 
expert  panel.  These  factors  vary  across  treatment  contexts 
and  must  be  adjusted  for  in  studies  comparing  outcomes 
across  treatment  sites.  Items  include  age  of  onset  of 
symptoms,  age  of  onset  of  aggression,  psychiatric  and 
substance  use  comorbidity,  mental  illness  and  substance 
abuse  history  of  parents,  parental  use  of  mental  health 
services,  recent  family  stressors,  housing  instability,  and 
socioeconomic  position  of  the  family.  Family  functioning, 
shown  to  be  a strong  determinant  of  treatment  success,  is 
measured  by  the  general  functioning  subscale  of  the 
McMaster  Family  Assessment  Device  (FAD:  Byles,  Byrne, 
Boyle,  & Offord,  1988). 

I Treatment  Amount  and  Quality 

There  is  a growing  consensus  that  mental  health  treatment 
of  adolescents  requires  multi-modal  approaches  (Baer  & 
i Nietzel,  1991).  This  view  is  partly  based  on  disappointing 
I results  from  clinical  studies  of  the  efficacy  of  single  agents. 

! Experience  throughout  the  country  with  integrated  systems 
of  care  suggests  that  a combined  therapeutic  program  is 
necessary.  Assessment  of  quality  and  amount  of  treatment  is 
based  on  reports  of  the  parent  and  a chart  review  guided  by 
a checklist  of  treatment  options.  Treatment  options  include: 

(a)  medications;  (b)  parent  training;  (c)  crises  services;  (d) 
in-home  services;  (e)  individual,  family,  and  group  therapy; 

(f)  case-management,  and  (g)  school-based  treatments. 
Frequency  of  each  intervention  is  documented.  Services 
utilization  materials  from  the  National  Adolescent  and  Child  i 
Treatment  Study  conducted  by  the  Research  and  Training 
Center  for  Children's  Mental  Health  (Silver  et  al.,  1992), 
and  the  Services  Use  in  Children  and  Adolescents  (Parent 
Self-Report)  instrument  developed  by  the  task  force  on 
outcomes  research  of  the  American  Academy  of  Child  and 
Adolescent  Psychiatry  were  adapted. 

I Pilot  Test  of  the  ATOM 

I Adolescents,  age  1 1 to  1 8,  from  both  the  inpatient  and 
! outpatient  units  at  Arkansas  Children's  Hospital  and  the 
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outpatient  units  at  Arkansas  Children's  Hospital  and  the 
Centers  for  Youth  and  Families  in  Little  Rock  were 
recruited.  Each  subject  and  his/her  parent  or  guardian 
completed  the  self-report  Baseline  Assessment  of  the 
ATOM.  The  adolescent  and  parent  completed  the 
Diagnostic  Interview  Schedule  for  Children  (DISC:  Version  i 
3.0;  Shaffer,  Schwab-Stone,  Fisher,  Cohen,  Paicenti,  Daves,  ; 
et  al.,  1993),  the  Child  Behavior  Checklist  (CBCL: 

Achenbach,  1991)  and  the  Columbia  Impairment  Scale 
I (Bird  et  al.,  1993)  to  allow  us  to  examine  the  relationships 
between  brief  module  measures  and  extensive  research 
measures  of  overlapping  constructs.  The  admitting  clinician 
completed  the  Clinician  Baseline  Assessment. 

Of  the  37  adolescents  referred  to  the  research  assistant,  3 1 
(84%)  agreed  to  participate  in  the  study.  Complete  baseline 
data  have  been  collected  for  all  enrolled  cases.  Subjects 
were  13.8  years  old  (sd  = 1.7)  on  average,  36%  female,  and 
39%  minority,  predominantly  African  American.  Common 
DISC  diagnoses  include  conduct  disorder  (23%),  depression 
or  dysthymia  (30%)  and  attention  deficit  hyperactivity 
disorder  (26%).  All  major  outcome  measures  in  the  module 
(symptom  severity,  functioning  in  the  home,  functioning  in 
the  school,  functioning  in  the  community,  functioning  with 
peers)  show  a distribution  approaching  normal  with  no 
evidence  of  a ceiling  or  floor  effect.  We  are  currently  in  the 
process  of  examining  correlations  between  module  and  gold 
standard  measures  of  comparable  constructs,  while 
examining  patterns  in  the  variation  between  adolescent  and 
parent  reports  on  outcomes.  Sentinel  indicators  show  that  i 
65%  of  subjects  have  been  suspended  from  school  for  one  | 
day  or  more;  40%  have  had  contact  with  the  police  in  the 
past  six  months;  and  1 5%  have  made  many  friends  who  are 
often  in  trouble  with  the  authorities.  Close  to  one-fifth  of 
parents  (17%)  reported  missing  days  at  work  because  of 
i their  child's  problems,  and  44%  found  the  household  routine  | 

I was  upset  because  of  the  adolescent's  problems. 

i On  average,  the  baseline  portion  of  the  module,  exclusive  of  i 
validating  instruments,  was  completed  in  25  minutes  (sd  = ' 

8.6)  by  adolescents,  28  minutes  (sd  = 7.8)  by  parents,  and  7 
minutes  (sd  = 9.4)  by  clinicians.  All  participants  filled  out  i 
the  module  with  a minimum  of  missing  data:  (a)  < 2%  from 
adolescents,  (b)  < 1%  from  parents,  and  (c)  < 5%  from 
clinicians.  Research  interviewers  began  six  month  follow-up  ; 
evaluations  with  patients  and  parents  in  July,  1996. 

Conclusions 

I 

Routine  outcomes  monitoring  requires  instruments  that  are 
comprehensive,  psychometrically  sound,  and  acceptable  to 
patients,  families,  and  agencies.  They  must  therefore  be 
brief,  inexpensive,  and  administered  by  non-clinicians  with 
only  minimal  training.  To  our  knowledge,  the  prototype 
ATOM  is  the  only  currently  available  battery  of  instruments 
that  measures  all  important  outcome  domains  easily  and 
economically.  As  such,  it  holds  great  promise  in  advancing 
routine  scientifically  rigorous  outcomes  monitoring  in 
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routine  scientifically  rigorous  outcomes  monitoring  in 
adolescent  mental  health  care. 
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Introduction 


Although  there  are  many  research  studies  evaluating  the 
efficacy  of  various  mental  health  interventions  for  children 
and  families,  there  are  very  few  reports  of  ongoing 
standardized  evaluation  of  treatment  outcomes  in 
community  mental  health  services  (Weisz  & Weiss,  1989; 
Weisz,  Weiss,  & Donenberg,  1992).  It  is  becoming 
increasingly  important  that  outcome  assessment  be  both 
widely  applied  and  linked  to  data  on  patient  characteristics 
and  service  utilization  (Sederer,  Hermann,  & Dickey,  1995). 
Community  mental  health  services  are  experiencing 
increased  internal  and  external  pressure  to  document  the 
efficacy  of  their  interventions  and  the  effectiveness  of  the 
service  system.  These  pressures  have  intensified  in  this  era 
of  managed  care  and  diminished  resources  for  mental  health 
services.  Furthermore,  these  pressures  dictate  that 
procedures  be  utilized  which  are  both  time  and  cost  efficient 
(Kurkland,  1995).  Unfortunately,  increased  pressure  to 
evaluate  outcomes  has  not  necessarily  coincided  with  the 
development  of  well  established  guidelines  for 
implementing  outcomes  evaluation.  Service  settings  are  left 
with  the  difficult  task  of  designing  their  own  protocols,  with 
or  without  consultation  from  researchers  who  have  the 
necessary  expertise  in  measurement  and  analysis. 

The  purpose  of  this  study  was  to  describe  the  process  of 
instituting  a standardized  mental  health  assessment  protocol 
in  a community  mental  health  treatment  program  serving 
maltreated  youth.  The  protocol  was  specifically  not 
implemented  as  a research  project,  but  rather  as  a low  cost 
component  of  standard  clinical  practice  at  the  clinic.  The 
goals  of  this  assessment  program  were  to; 

• Provide  descriptive  data  regarding  the  population 
utilizing  the  service,  and  describe  individual  and 
group  differences  in  this  population 

• Examine  the  efficacy  of  therapeutic  interventions  and 
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provide  a mechanism  for  monitoring  quality 
assurance 

• Supply  individual  clinical  data  to  mental  health 
providers  for  use  in  treatment  planning  and 
monitoring. 

Method 

Service  Site  and  Sample 

Children's  Hospital  Center  for  Child  Protection  (CCP) 
Mental  Health  Treatment  Program 

The  CCP  mental  health  treatment  program  is  an  outpatient 
therapy  program  specifically  for  children  and  adolescents 
who  have  experienced  maltreatment  in  the  form  of  sexual, 
physical,  or  emotional  abuse,  or  neglect.  The  majority  of 
patients  are  referred  for  sexual  abuse.  Individual,  group,  and 
family  treatment  are  available  from  an  interdisciplinary 
team  of  psychologists,  social  workers,  counselors,  and 
trainees.  In  the  fiscal  year  1994,  the  program  served  more 
than  250  children  and  their  parents.  A standardized 
assessment  protocol  was  introduced  in  mid  1 994  and  data 
for  242  subjects  have  been  collected. 

Description  of  Sample 

Sixty-two  percent  of  the  subjects  are  female;  the  mean  age 
was  8.7  years  (SD  = 4.0;  range  = 2-19).  Approximately  half 
of  the  sample  was  Caucasian  (47%);  African  Americans  and 
Hispanics  composed  20%  each  of  the  sample,  with  the 
remainder  classified  as  "Other."  The  majority  of  the  families 
(61%)  had  total  gross  incomes  of  less  than  $20,000. 
Approximately  two  thirds  of  the  children  were  referred 
primarily  because  of  sexual  abuse.  Six-month  follow-up 
data  has  been  more  difficult  to  obtain,  but  is  currently 
available  for  59  families. 

Measures 

Table  1.  below,  presents  the  measures  selected  for  the 
assessment  program,  by  domain  of  assessment  and 
informant. 

Issues  which  were  addressed  in  the  selection  of  instruments 
included: 

• Goals  and  objectives  of  intervention  (i.e.  relevance  of 
measure) 

• Psychometric  properties  of  instruments  for  the 
population  being  served  (e.g.,  SES,  race/ethnicity, 
culture,  age,  etc.),  and 

• Pragmatic  issues,  such  as  time  of  administration, 
costs,  personnel,  and  training  required. 

Procedure 

Assessment  measures  were  administered  by  graduate 
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Assessment  measures  were  administered  by  graduate 
students  and  research  assistants  who  received  training  in  the 
administration  of  the  standardized  instruments  and  some 
limited  clinical  interviewing  skills.  Families  were  scheduled 
for  an  intake  assessment  prior  to  initiating  treatment.  Data 
regarding  the  assessment  were  used  to  assign  families  to 
specific  types  of  treatment  and  providers.  A summary  of  the 
assessment  data  was  provided  to  the  clinician  prior  to  the 
family's  first  visit.  Follow-up  interviews  were  scheduled  six 
months  after  the  intake  assessment  and  the  same  battery  of 
instruments  was  repeated. 

Results 

Results  are  presented  and  discussed  based  on  the  relative 
success  of  achieving  the  program  goals  and  the  obstacles 
confronted  within  the  process. 

Goal  #1:  Provide  descriptive  data  regarding  the 
population  utilizing  the  services. 

A great  deal  of  descriptive  data  has  been  collected  on  the 
242  youth  assessed  upon  entering  the  mental  health 
treatment  program.  Such  data  includes  demographics, 
maltreatment  history,  mental  health  service  use  history, 
treatment  expectancies,  and  standardized  data  regarding 
mental  health  and  family  problems  and  competencies. 

There  are,  however,  limitations  in  the  usefulness  of  these 
data;  variability  in  the  clinical  staffs  comfort  and  familiarity 
with  the  instruments  jeopardizes  the  reliability  of  results. 
Also,  the  data  may  not  be  representative  of  all  youth  and 
their  families  utilizing  the  service  due  to  the  difficulty 
involved  in  successfully  assessing  all  children  and  their 
parents  prior  to  treatment  (Boren  et  al.,  1996).  Only 
approximately  60%  of  those  who  began  treatment  were 
assessed.  Anecdotal  data  suggests  that  the  reasons  for 
missing  the  intake  assessment  include,  crisis  intervention 
implemented  prior  to  assessment,  scheduling  difficulties, 
and  language  differences. 

Goal  #2:  Examine  the  efficacy  of  the  therapeutic 
interventions  and  provide  a mechanism  for  quality 
assurance 

The  utility  of  the  assessment  program  for  evaluating 
outcomes  obviously  relies  on  the  ability  to  collect  follow-up 
data,  as  well  as  on  the  representative  coverage  of  the 
assessments,  as  discussed  above.  To  date,  follow-up  data 
have  been  collected  for  only  59  youth  and  their  families,  too 
few  to  provide  a meaningful  assessment  of  treatment 
outcomes.  Uncertainty  regarding  whether  a true 
cross-section  of  the  patients  is  being  considered  in  outcome 
evaluation  may  arise  due  to  a lack  of  full  participation  in 
follow-up  interviews.  This  is  largely  the  result  of  difficulty 
in  tracking  some  families  after  their  initial  interview. 

A number  of  additional  obstacles  to  obtaining  reliable 
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A number  of  additional  obstacles  to  obtaining  reliable 
follow-up  data  are  worth  addressing.  One  such  obstacle, 
related  to  clinicians'  variable  familiarity  with  the 
instruments,  is  the  potential  for  variable  commitment  of  the 
treatment  program  staff  to  standardized  assessment.  The 
requirements,  for  example,  of  standardization  may  run 
counter  to  the  intentions  of  many  clinical  providers  to  adapt 
to  the  diverse,  idiosyncratic,  and  changing  needs  of 
consumers,  thereby  seeming  incongruent  or  even 
contradictory  to  the  clinical  and/or  administrative 
procedures  of  a mental  health  treatment  program.  Indeed, 
lack  of  consistent  commitment  could  impact  the  scheduling 
of  intake  interviews  and  follow-ups,  further  compromising 
the  universality  of  outcome  data  collection. 

Due  to  their  integral  role  in  the  success  of  such  an 
assessment  protocol,  clinicians  must  be  highly  involved  in  a 
collaborative  implementation  of  it.  The  researchers 
introducing  the  procedure  must  emphasize  their  role  as 
consultants,  and  must  transfer  administrative  functions  to 
the  clinic  in  a timely  manner.  Training  and  data  analysis  can 
remain  the  researchers'  responsibility;  however,  it  is 
imperative  to  involve  clinicians  early  in  order  to  establish  it 
as  a standard  clinical  practice,  rather  than  a research  project. 
This  entails  getting  input  from  clinical  staff  regarding 
research  questions  and  constructs  to  measure  prior  to  the 
protocol's  implementation. 

This  assessment  program  was  intentionally  implemented  as 
an  integral  component  of  ongoing  clinical  practice,  as 
opposed  to  a discrete  research  project.  Thus,  there  was  no 
attempt  to  implement  an  experimental  research  design  with 
random  assigmnent  to  treatment  conditions,  etc.  While  this 
approach  offered  clinical  advantages,  there  are  limitations  in 
the  interpretations  of  results  due  to  the  lack  of  standardized 
conditions. 

Goal  #3:  Supply  individual  clinical  data  to  mental  health 
providers  for  use  in  treatment  planning  and  monitoring 

Clinicians  have  reported  satisfaction  with  the  assessment 
summaries,  although  the  treatment  program  manager  reports 
that  the  summaries  are  not  frequently  used  for  assignment  to 
treatment.  Thus,  possible  variability  in  clinical  utility  is  an 
issue  of  which  to  be  aware. 

Changes  in  the  data  reporting  methods  have  been  instituted 
due  primarily  to  reimbursement  issues.  Reimbursement 
issues  arose  because  the  assessment  protocol  did  not  match 
the  categories  allowed  for  reimbursable  expenses  in  many 
public  and  privately  funded  managed  care  programs.  The 
procedures  were  therefore  changed,  and  made  more  costly, 
so  that  the  requirements  for  reimbursement  could  be  met. 
Instead  of  an  assessment  summary  completed  by  a 
paraprofessional,  a full  psychological  evaluation  report  was 
produced  by  a licensed  clinical  psychologist,  and  the 
assessment  was  billed  as  psychological  testing.  This  change, 
in  some  respects,  was  contrary  to  the  initial  purpose  of  the 
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in  some  respects,  was  contr^  to  the  initial  purpose  of  the 
protocol,  which  was  to  provide  a low  cost  assessment 
program  in  the  implementation  of  such  a protocol. 

Discussion 

It  is  feasible  to  implement  a standardized  assessment 
protocol  at  relatively  low  cost  in  a community  mental  health 
treatment  program.  There  are,  however,  several  obstacles 
that  need  to  be  addressed  including,  lack  of  full  coverage, 
reimbursement  requirements,  and  tracking  of  families  for 
follow-up.  The  introduction  of  standardized  assessment  to  a 
treatment  program  requires  the  following: 

• Early  involvement  and  training  of  staff; 

• Planning  for  fiscal  impact  and  flexibility  in 
reimbursement  plan; 

• The  selection  of  feasible  measures  (e.g.,  time,  cost, 
and  training  efficient),  the  data  from  which  will  meet 
the  goals  of  the  intervention; 

• Well  developed  database  and  client  tracking  systems; 

• Linkage  of  assessment  data  to  service  use  data;  and 

• Timeline  of  implementation  and  plans  for  transition 
from  trial  to  standard  clinical  practice. 

With  these  issues  addressed,  a successful  standardized 
assessment  protocol  can  be  implemented  with  benefits  to 
clinicians,  administrators,  and  researchers  alike. 
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Introduction 

Family-centeredness  is  often  listed  among  desirable 
elements  of  programs  for  children  with  special  needs  and 
their  families.  Validated  instruments  to  measure 
family-centered  care,  however,  have  been  lacking.  The 
author  and  Christopher  Petr  have  developed  and  validated 
an  instrument  that  measures  the  family-centeredness  of 
professional  behavior.  The  Family-Centered  Behavior  Scale 
(FCBS)  and  its  companion  tool,  the 

FCBS&endash;Importance,  are  completed  by  the  parents  or 
guardians  of  children  with  special  needs.  On  the  FCBS, 
parents  rate  how  often  a professional  with  whom  they  work 
performs  certain  family-centered  behaviors.  On  the 
FCBS-Importance,  parents  rate  the  importance  of  each  of 
these  behaviors. 

408 


Method 


Allen  & Petr  (1995b)  conducted  a cross-disciplinary  review 
of  literature  pertaining  to  family-centered  service  delivery. 
Based  on  this  review,  they  proposed  the  following 
definition;  Family-centered  service  delivery,  across 
disciplines  and  settings,  recognizes  the  centrality  of  the 
family  in  the  lives  of  individuals.  It  is  guided  by 
fully-informed  choices  made  by  the  family  and  focuses  on 
the  strengths  and  capabilities  of  families. 

The  authors  generated  a series  of  measurement  scale  items 
that  reflected  service  delivery  issues  identified  in  the 
literature  and  the  definition  above.  Focus  groups  of  parents, 
professionals,  and  the  project's  advisory  committee 
reviewed  this  collection  of  items.  The  draft  of  the  resulting 
measurement  instrument  was  field  tested  by  2 1 parents  of 
children  with  emotional/behavioral  disturbances. 

The  final  validation  edition  of  the  scale  had  32  items 
evaluating  staff  members'  performance  and  a matching  set 
of  items  on  which  parents/guardians  rated  the  importance  of 
each  item.  A satisfaction- with-this-staff-member  question 
and  several  demographic  questions  were  included  in  the 
research  package,  which  was  printed  in  both  English  and 
Spanish.  Half  of  the  surveys  asked  parents  to  describe  their 
best  staff  member,  while  the  other  half  asked  parents  to 
evaluate  their  worst. 

Copies  of  the  validation  edition  of  the  scale  were  sent  to 
about  1700  households  that  included  a child  with  special 
needs.  The  distribution  lists  for  the  surveys  were  collected 
from  organizations  that  dealt  with  a variety  of  special  needs 
(i.e.,  mental  health,  developmental  disabilities,  and  chronic 
health  problems).  Each  household  was  sent  two  surveys  in 
an  attempt  to  increase  the  response  rate  of  male  caregivers. 

Survey  Participants 

A total  of  443  parents/caregivers  of  406  different  children 
with  special  needs  returned  surveys  with  enough  data  to  be 
included  in  the  analysis.  About  87%  (N  = 379)  were  female; 
the  average  age  was  41  years.  Nearly  85%  (N  = 376)  were 
biological  parents  of  the  child  with  special  needs.  Most  of 
the  parents  were  White  (73%,  N = 323),  16%  (N  = 71)  were 
African-  American,  and  7%  (N  = 32)  were  Hispanic/Latino. 
Of  the  parents/caregivers,  71%  (N  = 313)  were  married, 

13%  (N  = 56)  were  divorced,  and  1 1%  (N  = 48)  had  never 
married.  Annual  household  income  averaged  around 
$35,000.  Responses  came  from  45  different  states. 

The  children  who  worked  with  staff  members  described  in 
the  survey  were  predominantly  male  (63%,  N = 252);  their 
average  age  was  10.7  years.  A total  of  68%  (N  = 275)  were 
White,  17%  (N  = 68)  were  African-American,  and  7%  (N  = 
29)  were  Hispanic/Latino.  Close  to  89%  (N  = 349)  lived 
with  their  parents.  Respondents  rated  the  severity  of 
disabilities  of  19%  (N  = 74)  of  the  children  as  mild,  45.5% 
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disabilities  of  19%  (N  = 74)  of  the  children  as  mild,  45.5% 
(N  = 177)  as  moderate,  and  35.5%  (N  = 138)  as  severe.  The 
most  prevalent  types  of  disability  were  developmental  delay 
(34%,  N = 138),  mental  retardation  (32%,  N = 128), 
emotional/behavioral  (29%,  N = 116),  and  learning 
disability  (29%,  N = 1 15). 

Of  the  returned  surveys,  40%  (N  = 178)  of  the  respondents 
described  their  "best"  staff  member,  33%  (N  = 148) 
described  their  "worst"  staff  member,  and  1 5%  (N  = 65) 
described  their  "only"  staff  member.  Most  of  the 
professionals  were  women  (75%,  N = 323);  their  average 
age  was  40  years.  Most  were  White  (82%,  N = 358);  10% 

(N  = 44)  were  African-American.  The  most  prevalent 
professions  represented  were  special  education  teacher 
(29%,  N = 1 19),  social  worker  (1 5%,  N = 62),  and  medical 
doctor  (12%,  N = 51).  The  most  prevalent  settings  in  which 
these  professionals  worked  were  education  (52%,  N = 201), 
health  (21%,  N = 80),  and  mental  health  (1 1%,  N = 42). 

Results 

Reliability  and  Validity 

Internal  consistency  of  scale  items  was  assessed  using 
Cronbach's  alpha  coefficient.  The  standardized  alpha  for  the 
32-item  scale  is  .97  (N  = 133).  The  test-retest  correlation  is 
,96(N=  128). 

Evidence  of  scale  validity  is  present  in  three  analyses:  the 
ability  of  the  scale  to  differentiate  between  "best"  and 
"worst"  staff  members,  correlations  between  scale  items  and 
satisfaction  with  the  staff  member,  and  importance  ratings 
of  respondents.  A comparison  of  means  among  the  best, 
worst,  and  only  staff  members'  scores  revealed  that  the 
means  of  all  32  items  were  different  at  the  .0014  level  or 
higher  (range:  F = 6.7199,  2,367,  p = .0014  to  F = 191.62, 
2,382,  p < .0000).  Differences  were  in  the  expected 
direction  of  poorer  family-centered  performance  by  "worst" 
staff  members  than  by  "best"  staff  members. 

The  researchers  hypothesized  that  families  who  receive 
more  family-centered  care  would  be  more  satisfied  with 
staff  members  that  work  with  them.  Zero-order  correlations 
between  items  on  the  32-item  scale  and  satisfaction  ranged 
from  .31  to  .81.  All  but  four  of  the  correlations  were  above 
.52.  All  correlations  were  significant  at  the  .000  level, 
supporting  the  hypothesis. 

Respondents  rated  all  32  behavior  items  as  important;  item 
means  ranged  from  3.2  to  4.7  (3  = important,  5 = extremely 
important).  The  three  most  highly-rated  importance  items 
pertained  to  the  staff  member  listening  to  the  family, 
treating  the  family  with  respect,  and  accepting  the  family  as 
important  team  members. 

Performance  of  Family-Centered  Behaviors 
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On  a scale  of  1 (Never)  to  5 (Always),  the  frequency  with 
which  staff  member  performed  the  32  family-centered 
behaviors  ranged  from  an  average  of  2.5  to  4.5.  For  the 
sample  as  a whole,  the  three  most  frequently  performed 
behaviors  were: 

• the  staff  member  does  not  make  negative  judgments 
about  us  because  of  ways  that  we  are  different  from 
the  staff  member; 

• the  staff  member  does  not  blame  me  for  my  child's 
problems;  and 

• the  staff  member  does  not  criticize  what  we  do  with 
our  child. 

The  three  least-often  performed  behaviors  were: 

• when  we  want  to  meet  other  families  that  have 
children  like  ours,  the  staff  member  helps  us  meet 
them; 

• the  staff  member  helps  my  family  get  services  from 
other  agencies  or  programs  as  easily  as  possible;  and 

• the  staff  member  helps  us  get  the  help  we  want  from 
our  family,  friends,  and  community. 

Scale  Revision 

The  researchers  wanted  to  reduce  the  size  of  the  scale  to 
make  it  more  user-friendly.  Six  items  were  eliminated  from 
the  scale,  two  because  they  performed  badly  in  several 
analyses  and  four  because  they  elicited  a relatively  large 
number  of  missing  responses  and  did  not  seem  to  make 
unique  contributions  to  the  scale.  The  alpha  coefficient  for 
the  resulting  26-item  Family-Centered  Behavior  Scale  (N  = 
206)  is  .98. 

Differences  for  Families  with  Children  who  have 
Emotional/Behavioral  Disabilities 

One-way  ANOVA  tests  were  used  on  the  final  26  scale 
items  to  explore  differences  between  scores  given  by  the 
subpopulation  of  caregivers  whose  children  had 
emotional/behavioral  problems  (EBD)  and  by  those  whose 
children  did  not  have  emotional/behavioral  problems 
(non-EBD).  The  EBD  subsample  rated  five  behaviors 
significantly  higher  in  importance  than  the  non-EBD 
subsample.  These  items  were: 

• the  staff  member  helps  my  family  get  services  from 
other  agencies  or  programs  as  easily  as  possible  (F  = 
15.70,  1,433,  p=. 0001); 

• the  staff  member  helps  us  get  the  help  we  want  from 
our  family,  friends,  and  community  (F  = 14.67,  1,431, 

p = .0001); 

• the  staff  member  helps  us  get  all  the  information  we 
want  and/or  need  (F  = 7.67,  1,435,  p < .01); 

• the  staff  member  cares  about  our  entire  family  (F  = 
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6.74,  l,433,p<.01);and 

• the  staff  member  accepts  our  family  as  important 
members  of  the  team  that  helps  our  child  (F  = 5.92,  1 , 
436,  p < .05). 

The  groups  had  similar  ratings  on  the  behaviors  performed 
most  frequently.  The  overall  scale  mean  for  the  EBD 
subsample  (3.23),  however,  was  significantly  lower  than  the 
non-EBD  subsample  (3.76;  F = 26.14,  1,439,  p < .0001). 

The  EBD  subsample  rated  every  behavior  as  being 
performed  less  frequently  than  the  non-EBD  subsample 
(range:  F = 5.80,  1,389,  p < .05  to  F = 34.00,  1,396,  p = 
.0000). 

The  researchers  also  examined  the  level  of  agreement 
between  the  importance  ratings  and  the  frequency  ratings. 
Difference  scores  were  computed  for  each  item  by 
subtracting  the  importance  score  from  the  frequency  score. 
Difference  scores  close  to  zero  indicate  a good  match 
between  consumers'  ratings  of  importance  of  a behavior  and 
the  frequency  with  which  it  is  practiced.  Positive  scores 
signify  that  frequency  exceeds  importance,  while  negative 
score  signify  that  frequency  is  below  importance.  Difference 
scores  for  the  EBD  subsample  were  significantly  lower  (less 
family-centered)  than  the  non-EBD  subsample  on  every 
behavior  (range:  F = 6.14,  1,380,  p ^ .05  to  F = 33.32,  1,420, 

p = .0000). 

Discussion 

The  Family-Centered  Behavior  Scale  can  be  used  to  inform 
service  delivery  systems  of  professional  behaviors  deemed 
important  by  caregivers  of  children  with  special  needs. 
Organizations  or  systems  that  deliver  services  to  children 
with  special  needs  and  their  families  may  use  the  FCBS  as 
part  of  their  assessment  of  the  extent  to  which  they  provide 
services  in  a family-centered  manner.  It  may  be  used  to 
inform  a program  of  its  particular  strengths  and  weaknesses 
in  terms  of  family-centeredness.  Plans  for  ways  to  improve 
on  the  areas  identified  as  weak  can  be  developed  and  carried 
out,  and  the  scale  then  be  administered  again  to  assess  the 
impact  of  intervention  on  service  delivery.  The  FCBS  also 
may  be  used  to  study  the  relationship  between 
family-centered  service  delivery  and  outcomes  that  are 
important  to  families  and  organizations.  Results  of  studies 
using  the  scale  may  be  helpfol  to  professional  training 
programs  that  wish  to  educate  providers  about 
family-centered  service  approaches. 

Results  of  this  study  suggest  that  professional  training  and 
system  reform  efforts  may  be  especially  important  for 
children  with  emotional/behavioral  disturbances  and  their 
families.  Caregivers  of  children  with  emotional  and 
behavioral  problems  in  this  sample  experience  their 
professional  interactions  as  less  family-centered  than  do 
parents  of  children  with  other  disabilities.  Family-centered 
behaviors  were  performed  less  frequently  with  the  EBD 
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subsample.  In  addition,  the  discrepancy  between  importance 
ratings  and  frequency  was  greater,  compounding  the 
negative  effects. 

The  FCBS  may  be  purchased  as  part  of  the  Family-Centered 
Behavior  Scale  and  User's  Manual  (Allen,  Petr,  & Brown, 
1995a)  from  the  Beach  Center  on  Families  and  Disability, 
The  University  of  Kansas,  3111  Haworth  Hall,  Lawrence, 
KS  66045. 
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A Brief  Multi-Dimensional  Children’s 
Level-of-Functioning  Tool 


Introduction 


Child  psychiatric  epidemiology  is  a rapidly  developing  field 
(e.g.,  Brandenburg,  Friedman,  & Silver,  1990;  Costello, 
1989;  Koot  & Verhulst,  1992),  and  with  it,  the  need  for 
adequate  assessment  tools  has  become  apparent.  Mental 
health  concerns  in  children  can  be  assessed  in  a variety  of 
ways  including:  (a)  psychiatric  diagnosis  using  the 
DSM-IV;  (b)  level-of- functioning  (LOF);  (c)  diagnosis  plus 
LOF;  and  (d)  exposure  to  risk  factors  for  mental  illness. 
Most  commonly,  diagnosis  plus  indication  of  LOF  is  used 
(Costello,  Bums,  Angold,  & Leaf,  1993).  However,  there  is 
a lack  of  adequate  LOF  instmmentation  for  youth. 


One  instmment  often  used  to  measure  LOF  is  the  Children's 
Global  Assessment  Scale  (CGAS;  Schaffer  et  ah,  1983). 

This  single-item  0-100  scale  was  designed  to  be  parallel  in 
stmcture  to  the  Global  Assessment  of  Functioning  scale  for 
adults  (GAF;  Endicott,  Spitzer,  Fleiss,  & Cohen,  1976) 
which  is  rated  as  axis  V of  a DSM  diagnosis.  Both  the 
CGAS  and  GAF  are  intended  to  be  rated  by  trained 
clinicians.  A weakness  of  uni-dimensional  scales  is  that  they 
provide  little  information  about  the  nature  and  complexity 
of  children's  mental  health  problems.  Some  youth  may  have 
rather  prominent  emotional/behavior  problems  in  one 
setting  (e.g.,  home/family,  school)  but  not  others.  A single 
rating  does  not  capture  such  subtleties.  Similarly,  certain 
domains  are  not  recognized  at  all  by  such  a scale  (e.g.,  legal, 
health).  Further,  only  multi-dimensional  scales  provide 
adequate  information  on  which  to  describe  clinically  similar 
groups  of  youth  that  can  provide  the  basis  for  designing 
services  and  service  packages  (e.g.,  Smukler,  Sherman, 
Srebnik,  & Uehara,  in  press). 

The  Child  and  Adolescent  Functioning  Assessment  Scale 
(CAFAS;  Hodges,  Bickman,  & Kurtz,  1991)  addresses 
some  of  the  weaknesses  of  uni-dimensional  LOF  scales.  The 
measure  assesses  five  domains  of  child  functioning:  role 
performance,  thinking,  behavior,  moods/emotions,  and 
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substance  use.  Caregiver  resources  are  also  rated.  Each 
domain  is  rated  on  a 4-point  severity  scale  with  behavioral 
anchors  for  each  rating.  This  scale  is  appealing  due  to  its 
rich  behavioral  descriptions  for  each  functioning  domain, 
and  it  has  been  used  in  large-scale  mental  health  services 
projects  (e.g.,  Fort  Bragg).  However,  extensive  training  of 
interviewers  is  needed  to  obtain  adequate  inter-rater 
reliability.  Furthermore,  the  interview  itself,  on  which 
ratings  are  based,  is  lengthy. 

Given  the  limitations  of  available  LOF  instruments  for 
children  and  adolescents,  development  of  a brief 
multi-dimensional  LOF  instrument,  that  requires  little  or  no 
training  to  be  rated  by  clinicians  as  well  as  parents,  would 
be  a valuable  contribution  to  the  field.  This  paper  presents 
data  describing  the  development  and  preliminary  data  on 
such  an  instrument,  the  "Ecology  Rating  Scale." 

Methods 

Participants 

The  study  collected  data  from  74  parents  of  children  (age 
4-1 8),  who  were  current  clients  at  one  of  six  community 
mental  health  centers  in  two  large  geographic  regions  of 
Washington  state.  The  regions  encompassed  rural,  suburban 
and  small  urban  centers.  Regions  were  selected  as  part  of  a 
larger  systems'  evaluation  assessing  the  impact  of  replacing 
mental  health  process  regulations  with  clinical  outcomes  as 
a method  of  accoimtability. 

The  sample  was  71%  male  and  13%  ethnic  minority. 
Ninety-three  percent  were  Medicaid-eligible.  Diagnoses 
were  primarily  adjustment  disorder  (43%),  attention  deficit 
(23%),  oppositional  defiant  disorder  (10%),  and 
depression/dysthymia  (10%).  The  sample  was  representative 
of  the  total  served  child  populations  of  the  selected  regions 
in  terms  of  gender,  age,  diagnostic  mix,  and  income  type. 
Participants  who  completed  baseline  surveys  were  contacted 
to  complete  assessment  materials  again  six  months  later. 

Case  managers  of  participating  child  clients  were  also  asked 
to  complete  a brief  assessment  of  client  functioning  and 
symptoms.  Of  the  74  participating  child  clients,  41  case 
manager  assessments  were  completed. 

Instruments 

Ecology  Rating  Scale.  The  Ecology  Rating  Scale  (ERS) 
was  designed  by  quality  assurance  and  children's  mental 
health  service  staff  at  a community  mental  health  center 
with  input  from  families  of  child  clients.  The  ERS  is  a brief, 
multi-dimensional  level-of-functioning  instrument  that  can 
be  rated  by  parents  or  clinicians.  The  ERS  is  comprised  of 
seven  domains  of  youth  functioning:  family,  school, 
emotional,  legal/Justice,  recreational,  health,  and  social  (see 
Figure  U.  For  each  domain,  respondents  are  asked  to 
"assess  the  degree  to  which  the  child's  life  is  influenced,"  in 
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"assess  the  degree  to  which  the  child's  life  is  influenced,"  in 
the  particular  domain,  by  problems  associated  with  his/her 
emotional^ehavioral  difficulties.  Each  domain  is  rated  on  a 
5-point  scale  of  severity  from  no  problems  to  severe 
problems  that  represents  that  degree  to  which  the  domain  is 
influenced  by  the  child's  emotional/behavioral  problems. 
Behavioral  anchors  are  written  on  the  scale  itself  for 
moderate  (3)  and  severe  (5)  levels. 

Conner's  Parent  Rating  Scale.  The  Conner's  Parent  Rating 
Scale  (Conners,  1970),  is  comprised  of  48  children's 
symptoms  and  behavior  problems,  each  rated  on  a 4-point 
severity  scale.  The  instrument  is  appropriate  for  use  with 
children  age  3 to  17  and  when  age-by-gender  norms  are 
available.  The  instrument  yields  five  factors:  conduct 
problems,  learning  problems,  psychosomatic  symptoms, 
impulsive-hyperactive  symptoms,  and  anxiety  symptoms. 
Adequate  reliability  and  validity  have  been  demonstrated 
with  this  instrument  (Conners,  1985). 

Children's  Global  Assessment  Scale.  For  participating 
child  clients,  case  managers  completed  a brief  functioning 
and  symptom  assessment  that  included  the  Ecology  Rating 
Scale  and  the  Children's  Global  Assessment  Scale  (CGAS; 
Shaffer  et  al,  1983)  described  above.  Studies  have 
demonstrated  adequate  inter-rater  reliability  as  well  as 
concurrent  and  discriminant  validity  for  the  CGAS  (Bird, 
Canino,  Rubio-Stipec,  & Ribera,  1987;  Shaffer,  et  al.,  1983; 
Steinhausen,  1987). 

Information  Systems  Data.  Data  gathered  from 
participating  agency's  information  systems  provided 
information  on  client's  socio-demographic  characteristics, 
residential  situation,  work/school  involvement,  and  service 
utilization. 

Results 

The  first  step  in  analysis  of  the  adequacy  of  the  ERS  was  to 
examine  the  distribution  of  scale  scores.  Scale  scores  were 
normally  distributed  with  the  exception  of  scores  for  legal 
and  health  scales,  domains  for  which  few  children 
demonstrated  difficulties.  Reliability  and  validity  were  then 
examined  and  findings  are  described  below. 

Reliability 

The  reliability  of  the  ERS  was  examined  with  respect  to 
both  internal  consistency  and  inter-rater  reliability.  While 
the  ERS  is  composed  of  seven  domains,  each  believed  to 
contribute  unique  information,  we  did  not  expect  domain 
ratings  to  be  completely  orthogonal.  Rather,  we  expected 
moderate  to  high  internal  consistency  for  the  scale  as  a 
whole.  Results  supported  this  expectation  with  alpha 
coefficients  of  .70  for  parent  ratings  (N  = 65)  and  .69  for 
clinician  ratings  (N  = 41). 

Inter-rater  reliability  was  tested  with  ERS  scores  from  27 
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Inter-rater  reliability  was  tested  with  ERS  scores  from  27 
clients  rated  by  two  clinicians  who  reportedly  knew  the 
clients  well.  Results  suggested  that  reliability  was 
potentially  adequate  for  family,  legal,  recreation,  and  social 
domains,  with  intra-class  correlations  of  .50,  .65,  .71.,  and 
.83  respectively.  Intra-class  correlations  for  the  remaining 
scales  were  less  than  adequate  and  are  as  follows:  school 
(.44),  emotions  (.37),  and  health  domains  (.38).  If  used  as  an 
overall  index  of  functioning,  however,  the  total  ERS  score 
demonstrated  strong  inter-rater  reliability  with  an  intra-class 
correlation  of  .75. 

Validity 

Concurrent  validity  was  first  examined  as  the  correlation 
between  clinician-rated  ERS  scores  and  the  CGAS  (N  = 41). 
Although  school  and  emotion  domains  demonstrated 
relatively  weak  inter-rater  reliability,  they  were  among  the 
scales  with  stronger  concurrent  validity  with  the  CGAS, 
with  correlations  of  .62  and  .54.,  respectively,  both 
significant  at  the  p < .001  level.  The  recreation  (r  = .49)  and 
social  scales  (r  = .45)  were  significantly  correlated  to  the 
CGAS  at  the  p < .01  level.  Family,  legal,  and  health 
domains  were  moderately,  though  non-significantly, 
correlated  with  the  CGAS  with  correlations  ranging  from  r 
= .27  to  .41 . The  total  ERS  score  was  strongly  and 
significantly  correlated  with  the  CGAS  (r  = .72,  p < .001). 

Correlations  between  parent-rated  ERS  scores  and  subscale 
scores  of  the  Conner's  Parent  Rating  Scale  were  conducted 
as  a second  means  of  obtaining  concurrent  validity 
information.  Predicted  relationships  are  shown  in  Table  1. 

Consistent  with  expectations,  parents'  perceptions  of 
problems  in  the  "family"  and  "emotions"  domains  were 
associated  with  externalizing  behaviors  such  as  the 
"impulsive-hyperactivity"  and  "conduct"  subscales.  Also,  as 
predicted,  problems  in  the  "school"  domain  were  associated 
with  learning  problems.  Health  problems  were  related  to 
anxiety,  consistent  with  expectations  regarding  the 
relationship  of  somatization  with  anxiety.  Social  and 
recreation  problems  were  associated  with  conduct 
difficulties.  This  relationship  is  reasonable  considering  that 
youth  with  conduct  issues  will  likely  have  difficulty  getting 
along  with  others  socially  and  in  recreational  settings  (as 
well  as  within  the  family).  Legal  problems  were  also 
predicted  to  be  associated  with  the  conduct  subscale;  and 
the  lack  of  association  could  be  due  to  restriction  of  range 
on  the  legal  domain  scale.  The  ERS  total  score,  as  rated  by 
parents,  appears  to  be  most  highly  related  to  externalizing 
behavior  problems  and  learning  problems,  rather  than 
internalizing  problems. 

Treatment  validity  was  tested  through  examination  of 
whether  ERS  scores  change  over  the  course  of  six  months  of 
outpatient  mental  health  treatment.  The  numbers  of 
individuals  who  demonstrated  change,  and  whether  the 
sample  change  was  significant,  according  to  t-test 
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sample  change  was  significant,  according  to  t-test 
comparisons  for  matched  samples  is  shown  in  Tables  2 and 
3. 

Table  2 shows  parent  ERS  ratings,  and  Table  3 shows  case 
manager  ERS  ratings.  The  number  of  individuals  showing 
improvement  relative  to  those  demonstrating  no  change  or 
declining  functioning  according  to  parents  is  greatest  for  the 
following  domains:  school,  emotion,  and  legal.  Case 
managers  also  reported  many  more  individuals  improving 
regarding  the  recreation  domain.  Overall  sample  change  on 
the  parent-rated  ERS  scales  was  significant,  and  in  the 
expected  direction,  for  two  domains.  Trends  in  the  expected 
direction  were  found  for  the  other  ERS  scales.  Significant 
change  was  found  on  all  but  the  "social"  domain  for  case 
manager-rated  ERS  scales. 

Discussion 

This  presentation  described  a brief,  multi-dimensional,  EOF 
instrument  for  youth,  the  Ecology  Rating  Scale  (ERS).  The 
ERS  assesses  functioning  across  7 life  domains:  school, 
emotions,  legal,  health,  recreation,  social,  and  family.  In 
contrast  to  extant  measures,  the  ERS  has  the  following 
unique  combination  of  attributes:  (a)  assesses  multiple 
domains  of  functioning,  (b)  requires  little  training,  and  (c) 
demonstrates  promising  psychometric  properties.  While  we 
do  not  advocate  use  of  any  EOF  tool  without  proper 
training,  the  psychometric  characteristics  of  the  ERS 
reported  here  were  found  without  clinicians  having  any 
training.  Ideally,  clinicians  using  such  instruments  would 
periodically  conduct  formal  or  informal  tests  of  inter-rater 
agreement  in  the  course  of  discussing  cases  among  clinical 
team  members. 

Study  findings  regarding  the  ERS  suggested  some  areas  of 
internal  consistency.  Inter-rater  reliability  was  strongest  for 
social,  recreation,  and  legal  subscales.  Other  scales  had 
weak  inter-rater  reliability,  perhaps  due  to  assessing 
behaviors  related  to  non-clinical  settings  (e.g.,  school)  or 
less  recognizable  internal  states  (e.g.,  emotions  and  health). 
Hodges  (1991)  also  found  weak  inter-rater  agreement  for 
scales  assessing  internal  states.  On  the  other  hand,  more 
behavioral  anchors  combined  with  rater  training  may 
improve  inter  rater  agreement  of  the  ERS. 

Concurrent  validity  was  demonstrated  for  ERS  parent 
ratings  with  the  Conner's  behavior  rating  scale.  Clinician 
ratings  on  ERS  scales  were  also  correlated  with  a 
uni-dimensional  EOF  measures,  the  Children's  Global 
Assessment  Scale.  With  the  exception  of  the  "legal" 
subscale  of  the  ERS,  all  ERS  scales  provided  some  evidence 
of  concurrent  validity.  Correlations  with  the  legal  subscale 
were  likely  affected  by  restriction  of  variance  of  responses 
on  the  subscale.  Evidence  of  treatment  validity  (i.e., 
significant  change  in  expected  direction  following  6 months 
of  treatment)  is  also  demonstrated  for  ERS  scales. 
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Further  development  of  the  ERS  is  needed.  In  addition  to 
developing  more  behavioral  anchors  for  scale  points,  norms 
for  the  scale  should  be  developed  on  a larger  sample  that 
includes  higher  rates  of  ethnic  minority  youth. 

Preliminary  testing  of  the  ERS  demonstrates  that  it  shows 
promise  as  a brief,  practical  tool  for  LOF  assessment  in 
youth  receiving  community  mental  health  services.  Uses  for 
such  a tool  include  clinical  outcome  assessment, 
performance  measurement,  continuous  quality 
improvement,  and  services  research. 
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Introduction 


In  1984,  the  U.S.  Congress  expressed  a growing  concern 
regarding  the  mental  health  status  of  children  and  youth 
with  serious  emotional  and  behavioral  disorders,  and  the 
system  responsible  for  serving  these  youth.  Under  the 
auspices  of  the  National  Institutes  of  Health,  Mental  Health 
Branch,  the  Child  and  Adolescent  Service  System  Program 
(CASSP)  was  established  to  promote  a comprehensive 
working  model  of  service  provision.  As  originally 
presented,  this  "model"  was  not  intended  as  a static  product, 
but  as  a set  of  guidelines  and  principles  in  service  of  a 
process.of  research-driven,  bottom-up  service  system 
improvement. 


Historically,  children  who  suffer  from  the  debilitating 
influences  of  serious  emotional  disturbances  have  not 
received  services  outside  very  restrictive  and  controlling 
settings,  and  very  little  meaningful  data  has  been  collected 
which  promotes  informed  treatment  or  public  policy 
judgments  or  decisions.  Most  critically,  there  exists  a 
serious  lack  of  data  regarding  how  to  best  serve  these  youth. 
Research  efforts  have  principally  focused  on  describing 
I these  youth  in  terms  of  gross  demographic  variables  and 

I service  usage. 

I 

I The  descriptive  research  approach  has  provided  some  basic 

I information  about  these  youth;  however,  it  has  not  provided 

I much  useful  information  regarding  the  relative  contribution 

I of  various  psychosocial  variables  in  the  process  of 

! community  integration  or  exclusion.  Identifying  the 

pathways  leading  to  community  exclusion,  based  upon 
g unique  youth  psychosocial  variables,  could  inform  both 

I treatment  and  policy  judgments  regarding  system  needs  and 

I improvements. 

I Purpose 
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These  data  are  part  of  an  evolving  project  designed  to 
investigate  two  related  questions:  (a)  What  youth  and 
adolescent  psychosocial  variables  are  associated  with 
community  exclusion?  and  (b)  Are  there  consistent 
pathways  of  exclusion  associated  with  those  psychosocial 
variables  or  clusters  of  variables? 

Method 

Subjects 

The  investigators  obtained  a complete  set  of  demographic 
data  on  thirty-five  adolescents  served  in  two  group  living 
programs.  The  group  living  programs  were  designed  to  fiilly 
integrate  these  adolescents  into  all  aspects  of  community 
life,  including  public  education.  The  adolescents  served  by 
these  two  programs  were  either  returned  to  the  state,  from 
out-of-state  restrictive  residential  treatment  centers,  or 
placed  in  the  group  living  programs  to  prevent  out-of-state 
placements.  These  youth  were  described  as  uncontrollable, 
demonstrating  disorders  of  conduct,  including  aggression, 
assault,  property  damage,  law  violation,  inappropriate 
sexual  behavior,  with  most  having  more  than  one  prior 
psychiatric  hospitalization. 

Measurement/Instrumentation 

In  an  attempt  to  identify  adolescent  behaviors  associated 
with  functional  problems  in  the  two  most  influential 
contextual  settings  in  which  youth  must  participate,  the 
following  instruments  were  employed: 

• Daily  Adjustment  Indicators  (DA\\^VLVchdixd,  1990). 
The  DAI  is  a 25  item  behavior  checklist  designed  to 
be  completed  by  adult  caretakers  on  a daily  basis.  In 
this  present  study,  monthly  aggregates  formed  the 
basic  unit  of  DAI  analysis.  The  score  for  the 
adolescents  during  the  second  month  on  each 
behavioral  item  was  used  as  a concurrent  and 
potentially  predictive  variable  with  the  Sutter-Eyberg 
scores  described  below.  The  second  full  month  of 
data  was  chosen  in  an  attempt  to  control  for  variations 
in  behavioral  responding  due  to  a new  living 
environment. 

• Sutter-Eyberg  Student  Behavior  Inventory  (SESBI; 
Sutter  & Eyberg,  1992).  The  SESBI  is  a 36  item 
behavior  checklist  that  allows  the  teacher  to  rate  each 
child's  behavior  from  never  to  always.  This  inventory 
also  rates  a second  dimension  by  asking:  Is  this 
[behavior]  a problem  for  you?  Teachers  completed 
this  checklist  every  other  month.  This  measure 
includes  a total  problem  score  and  a total  intensity  of 
behavior  score  for  each  adolescent.  In  order  to 
maintain  temporal  consistency  with  the  DAI  data 
collection,  each  adolescent's  primary  teacher 
completed  this  checklist  at  the  end  of  the  second 
month  of  school.  Total  "problem"  and  "intensity" 
scores  formed  the  basic  unit  of  analysis. 
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scores  formed  the  basic  unit  of  analysis. 

• Community-Oriented  Programs  Environment  Scale 
(COPES;  Moos,  1987).  This  instrument,  designed  for 
community-based  treatment  programs,  includes  100 
items  and  10  subscales.  Youth  rate  their  living 
environment  every  month.  Again,  total  scores 
associated  with  the  second  month  formed  the  basic 
unit  of  analysis. 

Results 

Table  1 shows  the  means  and  standard  deviations  as  well  as 
the  correlations  between  the  demographic  variables  and  the 
SESBI  scores.  Of  these,  only  the  number  of  prior 
hospitalizations  was  significantly  correlated  with  the  SESBI 
scores. 

Table  2 shows  the  univariate  statistics  for  and  correlations 
between  the  components  of  the  DAI  and  the  SESBI.  Six  of 
the  components  were  significantly  correlated  with  both  of 
the  SESBI  scores  (marked  with  *).  These  six  were 
aggregated  into  a composite  score,  and  the  correlations 
between  this  composite  score  and  the  SESBI  are  shown  near 
the  bottom  of  Table  2. 

Knowing  that  the  number  of  prior  hospitalizations  was  the 
only  demographic  variable  correlated  with  the  SESBI 
scores,  and  that  these  scores  were  also  correlated  with  the 
DAI  Composite  score,  a series  of  multiple  regression 
analyses  were  completed  to  examine  the  incremental  utility 
of  the  DAI  composite  score,  and  to  explore  the  possibility  of 
an  interaction  between  the  DAI  composite  score  and 
selected  demographic  variables  (e.g.,  number  of  prior 
hospitalizations,  gender,  age,  and  the  COPES). 

Table  3 shows  the  results  of  a hierarchical  regression 
analysis  used  to  examine  the  incremental  utility  of  the  DAI 
composite  score  relative  to  the  number  of  prior 
hospitalizations,  and  whether  there  was  an  interaction 
between  these  two  variables  as  they  relate  to  the 
Sutter-Eyberg  Problems  score.  As  that  table  shows,  the  R 
square  of  the  two-predictor  model  was  significantly  larger 
than  of  the  model  including  only  the  number  of  prior 
hospitalizations,  indicating  that  the  DAI  composite  score 
has  independent  predictive  utility.  Further,  there  is  a 
significant  interaction  of  these  variables,  as  shown  in  Step  3 
of  the  analysis.  Figure  1 represents  the  form  of  this 
interaction,  by  plotting  the  simple  regression  line  of  the  DAI 
composite  onto  Sutter-Eyberg  Problems  for  three  different 
values  of  the  number  of  prior  hospitalizations,  at  one 
standard  deviation  above  the  mean  of  priors,  at  the  mean  of 
priors,  and  at  one  standard  deviation  below  the  mean  of 
priors.  As  the  figure  shows,  the  DAI  composite  score  is  a 
better  predictor  of  Sutter-Eyberg  Problems  for  clients  with 
higher  numbers  of  prior  hospitalizations  (the  regression 
slope  is  not  different  from  0.0  for  the  low  priors).  Parallel 
analyses  for  the  Sutter-Eyberg  intensity  showed  that  while 
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inclusion  of  the  DAI  composite  scores  did  increase  the  fit  of 
the  model  above,  when  only  the  number  of  prior 
hospitalizations  was  included  (R  squared  increased  from  . 1 0 
to  .55,  p < .001),  there  was  no  interaction  of  priors  and  the 
DAI  for  this  criterion  variable. 

Table  4 shows  the  results  of  a hierarchical  regression 
analysis  used  to  examine  whether  there  was  an  interaction 
between  gender  and  the  DAI  composite  score,  as  they  relate 
to  the  Sutter-Eyberg  Problem  score.  As  that  table  shows 
(Step  3),  there  is  a significant  interaction  of  these  variables. 
Figure  2 represents  the  form  of  this  interaction,  by  plotting 
the  simple  regression  line  of  the  DAI  composite  score  onto 
Sutter-Eyberg  Problems  for  males  and  females  as  the  figure 
shows,  the  DAI  composite  score  is  a better  predictor  of 
Sutter-Eyberg  Problems  for  males  (the  regression  slope  is 
not  different  from  0.0  for  females).  Parallel  analyses  for  the 
Sutter-Eyberg  Intensity  showed  that  while  inclusion  of  the 
DAI  composite  score  did  increase  the  fit  of  the  model  above 
that  including  only  the  number  of  prior  hospitalizations  (R 
square  increased  from  .03  to  .44,  p < .001),  there  was  no 
interaction  of  Priors  and  the  DAI  for  this  criterion  variable. 

Additional  multiple  regression  analyses  revealed  that  there 
was  no  interaction  between  the  DAI  composite  and  age  nor 
the  COPES  score,  for  either  Sutter-Eyberg  Intensity  or 
Problems. 

Conclusions 

These  results  demonstrate  that  while  demographic  variables 
have  little  utility  to  predict  which  adolescents  will  be 
excluded  from  community  setting  (only  the  number  of  prior 
hospitalizations  was  correlated  with  the  criterion  variables), 
several  items  from  the  DAI  were  useful.  A composite  of 
these  items  (physical  aggression,  verbal  aggression,  sexual 
acting-out,  anxious  behavior,  and  police  contact)  was 
substantially  correlated  with  both  the  Sutter-Eyberg 
Intensity  and  Problem  scores. 

Further,  with  respect  to  the  SESBI  score,  this  DAI 
Composite  showed  interactions  with  both  gender  and 
number  of  prior  hospitalizations.  The  patterns  of  these 
interactions  raised  the  possibility  that  different  patterns  of 
behavior  may  serve  as  better  predictors  for  females  than 
males,  and  for  those  with  fewer  prior  hospitalizations. 

Based  on  these  findings,  we  would  like  to  encourage  the 
further  exploration  of  the  utility  of  the  DAI  and  the  SESBI 
for  studying  this  population.  These  behavior-observational 
instruments  appear  to  have  potential  to  help  identify  the 
pathways  associated  with  community  exclusion  of  youth 
with  serious  emotional  or  behavioral  disturbances. 
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Introduction 

Kaleidoscope,  Inc.,  in  Chicago,  Illinois,  is  a child  welfare 
agency  that  provides  unconditional,  individualized,  and, 
when  necessary,  wraparound  services  to  over  600  children 
and  families.  The  purpose  of  Kaleidoscope  is  to  support  or 
recreate  the  family  by  helping  to  develop  or  redevelop  the 
love,  care,  and  discipline  of  its  members.  The  Satellite 
Family  Outreach  Program  is  one  of  the  programs  offered  by 
Kaleidoscope.  The  primary  goal  of  the  Satellite  Program  is 
to  help  maintain  referred  children,  parents,  and  siblings  as 
an  intact  family  unit.  The  families  served  by  Satellite 
experience  numerous  difficulties  and  needs.  Most  are  single 
head-of-household  families  functioning  on  incomes  below 
the  poverty  level.  Many  have  open  cases  with  Illinois'  child 
protection  agency. 


Although  there  is  much  interest  in  the  methods  child  welfare 
agencies  such  as  Kaleidoscope  use  in  aiding  their  clients, 
there  is  very  little  research  concerning  the  specific  activities 
performed  by  the  staff  members  of  such  agencies.  The 
present  study  examines  (a)  the  types  of  activities  performed 
by  the  Satellite  staff,  (b)  the  frequency  of  the  occurrence  of 
these  activities,  (c)  with  whom  the  staff  member  were 
engaged  in  each  activity,  (d)  where  each  activity  occurred, 
and  (e)  the  duration  of  each  activity.  The  purpose  of  this 
study  was  to  investigate  how  the  clients  of  the  Satellite 
Family  Outreach  Program  are  served  by  the  social  workers 
and  family  workers  of  Satellite. 


Method 

Setting 

This  study  was  conducted  at  Kaleidoscope,  Inc.,  a 
non-profit  child  welfare  agency  in  Chicago,  Illinois.  In 
January  1994,  Kaleidoscope  received  a grant  from  the  Annie 
E.  Casey  Foundation  to  evaluate  its  Satellite  Family 
Outreach  Program.  This  study  is  one  component  of  the 
overall  evaluation.  The  Satellite  program  serves 
approximately  48  families  at  any  one  time  and  employs  a 
staff  of  four  administrators,  three  supervisors,  four  social 
workers,  and  16  family  workers. 

Participants 

The  Professional  Activities  Checklist  (PAC)  was  completed 
at  two  time  intervals:  Time  1 (May/June  1995)  and  Time  2 
(September/October  1 995).  At  Time  1,17  staff  completed 
the  checklist.  Twenty-four  percent  (N  = 4)  of  the 
respondents  were  social  workers  and  76%  (N  = 13)  were 
family  workers.  Time  1 participants  ranged  from  23  to  54 
years  of  age.  The  mean  age  was  32.4  years.  A total  of  71% 
(N  = 12)  of  the  Time  1 participants  were  female,  and  29% 

(N  = 5)  were  male.  The  racial  composition  was  53%  African 
American  (N  = 9),  41%  Caucasian  (N  = 7),  and  6%  Asian 
American  (N  = 1).  At  Time  2,  17  staff  participated,  four 
were  social  workers  (24%)  and  13  were  family  workers 
(76%).  The  average  age  was  32.9  years,  ranging  from  23  to 
54  years.  Seventy-one  percent  (N  = 12)  of  the  respondents 
were  female;  29%  (N  = 5)  were  male.  Forty-seven  percent 
(N  = 8)  were  African  American,  41%  (N  = 7)  were 
Caucasian,  and  12%  (N  = 2)  were  Asian  American.  Thirteen 
staff  members  participated  at  both  Time  1 and  Time  2. 

Instrument 

The  PAC  was  developed  in  three  steps.  First,  evaluation 
team  members  met  with  a group  of  five  Satellite  staff 
members.  The  staff  members  were  asked  what  activities 
they  perform,  with  whom  they  perform  them,  and  where 
they  perform  them.  Based  on  the  staff  responses,  the 
evaluators  constructed  a first  draft  of  the  PAC.  Next,  the 
evaluators  met  again  with  the  group.  Staff  members 
provided  feedback  on  the  first  draft  and  evaluators 
incorporated  the  appropriate  changes  into  the  PAC.  Then, 
several  staff  members  completed  the  PAC  daily  for  three 
consecutive  days  in  a pilot  study.  Based  on  this  feedback, 
evaluators  and  staff  noted  any  remaining  inconsistencies, 
and  the  final  version  of  the  Professional  Activities  Checklist 
was  then  constructed. 

The  instrument  consists  of  19  professional  activities  in 
which  Satellite's  family  workers  and  social  workers  engage 
(e.g.,  individual  counseling,  advocacy).  For  each  day,  the 
worker  indicates  how  many  times  each  activity  is  performed 
(i.e.,  the  frequency).  For  each  occurrence  the  worker 
indicates  with  whom  the  activity  is  performed  (e.g.,  referred 
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indicates  with  whom  the  activity  is  performed  (e.g.,  referred 
client,  other  family  members),  where  the  activity  is 
performed  (e.g.,  client's  home,  community),  and  how  long 
the  activity  lasts.  The  checklist  takes  about  10  minutes  to 
complete. 

Procedure 

As  stated,  the  study  was  conducted  at  two  time  intervals 
(Time  1 and  Time  2).  Each  time  interval  spanned  a five 
week  period.  Satellite  staff  are  organized  into  teams 
comprised  of  one  social  worker  and  four  family  workers. 

One  staff  member  from  each  team  completed  the  PAC  daily 
for  one  of  the  five  weeks  in  the  time  interval.  Staff  members 
were  given  the  PAC  forms  by  an  evaluation  team  member 
five  days  prior  to  their  assigned  week.  The  evaluation  team 
member  gave  participants  a reminder  call  on  the  first  day 
that  they  were  to  complete  the  checklist.  At  the  end  of  their 
assigned  week,  the  participants  returned  the  checklists  to  the 
evaluation  team.  If  a participant  was  sick  or  not  working  on 
one  of  their  assigned  days,  they  completed  the  PAC  for  the 
same  day  the  following  week. 

Results 

To  identify  the  professional  activities  of  the  Satellite  staff, 
the  frequencies  per  day  per  staff  member  and  the  time  spent 
per  day  per  staff  member  were  examined.  Findings  are 
presented  for  both  the  fi'equencies  and  time  spent  for  both 
time  intervals  (Time  1 and  Time  2)  and  the  differences 
between  Time  1 and  Time  2 in  minutes  spent  per  day  on 
each  activity. 

At  Time  1 , the  most  frequently  occurring  professional 
activities  were  writing  reports  (1.51  per  day),  individual 
counseling  (1.05  per  day),  and  collateral  contacts  (1.00  per 
day).  The  least  frequent  activities  were  crisis  intervention 
(0.02  per  day),  training  (0.04  per  day),  relationship 
counseling  (0.08  per  day),  and  employment/vocational  (0.08 
per  day).  Staff  spent  the  most  time  writing  reports  (111  .97 
minutes  per  day),  counseling  individuals  (52.58  minutes  per 
day),  in  meetings  (49.12  minutes  per  day),  making  collateral 
contacts  (44.42  minutes  per  day),  and  in  transportation 
(39.01  minutes  per  day).  Staff  spent  the  least  amount  of  time 
in  crisis  intervention  (1.41  minutes  per  day),  food 
depository  (2.12  minutes  per  day),  and  housing  (2.29 
minutes  per  day). 

At  Time  2,  the  most  frequently  occurring  professional 
activities  per  day  were  written  reports  (1 .38  per  day), 
collateral  contacts  (1.08  per  day),  counseling  individuals 
(0.92  per  day)  and  transportation  (0.87  per  day).  The  least 
frequently  occurring  professional  activities  per  day  were 
employment/vocational  (0.00  per  day),  relationship 
counseling  (0.04  per  day),  housing  (0.04  per  day),  and 
training  (0.05  per  day).  Staff  spent  the  most  time  in  writing 
reports  (1 1 1 .60  minutes  per  day),  meetings  (5 1 .25  minutes 
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per  day),  transportation  (37.39  minutes  per  day),  counseling  | 
individuals  (35.52  minutes  per  day),  and  collateral  contacts 
(36.47  minutes  per  day).  Staff  spent  the  least  amount  of  time 
in  employment/vocational  (0.00  minutes  per  day), 
relationship  counseling  (0.89  minutes  per  day),  housing 
(0.89  minutes  per  day),  food  depository  (1.61  minutes  per 
day),  and  resource  referral  (2.86  minutes  per  day).  A 
complete  list  of  the  frequencies  and  the  time  spent  on  each 
activity  at  both  Time  1 and  Time  2 is  displayed  in  Table  1. 

Across  the  two  time  periods,  there  were  very  few 
differences.  Activities  that  showed  a substantial  increase  in 
time  spent  per  day  (i.e.,  more  than  four  minutes)  were 
family  counseling  (1 1.67  minutes),  training  (10.99  minutes), 
family  visits  (10.43  minutes),  and  advocacy  (6.08  minutes). 
Activities  that  showed  a substantial  decrease  (i.e.,  more  than 
four  minutes)  in  time  spent  per  day  were  counseling 
individuals  (-17.06  minutes)  and  collateral  contacts  (-7.65 
minutes).  The  other  1 3 activities  were  relatively  stable 
across  the  two  occasions.  A complete  list  of  the  changes  in 
time  spent  between  Time  1 and  Time  2 is  shown  in  Table  2. 

Discussion 

The  findings  of  this  study  are  useful  in  a number  of  ways. 
First,  Kaleidoscope  can  examine  the  findings  and  determine 
whether  the  activities  the  staff  engage  in  accurately  reflect 
the  mission  of  the  Satellite  Family  Outreach  Program.  For 
instance,  one  goal  of  the  Satellite  Program  is  to  help 
families  stay  together  through  intensive  case  management. 
Therefore,  program  administrators  can  use  the  Professional 
Activities  Checklist  data  to  determine  how  much  time  the 
workers  are  spending  with  the  clients  and  whether  this 
amount  of  time  adequately  meets  the  goal  of  intensive  case 
management. 

Second,  the  information  indicates  what  types  of  services  are 
used  most  frequently  by  clients.  This  outcome  may  enable 
the  agency  to  further  develop  services  tailored  to  the 
specific  needs  of  their  clients.  For  instance,  as  individual 
counseling  and  recreational  activities  are  among  the  services 
most  used  by  clients,  program  administrators  may  consider 
providing  counseling  training  to  staff  members  and 
incorporating  counseling  into  recreational  activities. 

Third,  the  data  on  the  professional  activities  of  staff 
members  may  reveal  some  important  information  when 
combined  with  other  research  on  the  same  staff  members. 

For  example,  combining  the  analysis  of  professional 
activities  with  research  on  job  satisfaction  (another  area  we 
are  currently  studying)  may  reveal  significant  relationships 
between  the  two  variables.  For  example,  those  who  are 
frequently  advocating  for  their  clients  may  have  a higher 
level  of  job  satisfaction  than  those  who  spend  their  time  in 
meetings,  or  vice  versa. 

Finally,  these  data  may  assist  other  agencies  interested  in 
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providing  wraparound-service  programs.  Specifically,  these 
data  outline  the  daily  activities  that  are  engaged  in  by  staff 
who  are  following  a wraparound  model  of  service  delivery; 
other  agencies  interested  in  following  this  model  of  care 
may  be  able  to  use  these  data  when  deciding  what  specific 
staff  activities  are  necessary  for  the  agency  and  its  clients. 
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Introduction 

Current  thinking  about  the  child  mental  health  system 
emphasizes  the  importance  of  consumer  satisfaction  and  a 
child-centered  approach  (Knitzer,  1982;  Stroul  & Friedman, 
1 986).  Successful  implementation  of  these  principles 
requires  a reliable  and  valid  method  of  producing 
differentiated  information  about  young  people's  satisfaction 
with  their  services.  Efforts  to  develop  such  methods  have 
only  recently  begun.  Existing  measures  have  been 
characterized  as  being  limited  by  a lack  of  psychometric 
information,  sampling  problems,  vulnerability  to  social 
desirability  response  bias,  and  a lack  of  differentiation 
among  dimensions  of  satisfaction  (Young,  Nicholson,  & 
Davis,  1995). 


The  Youth  Client  Satisfaction  Questionnaire  (YCSQ)  was 
designed  to  differ  from  existing  measures  of  this  construct 
by  combining  several  characteristics.  First,  this  instrument 
is  administered  as  a telephone  interview,  which  provides  a 
higher  response  rate  than  mailed  surveys  (Young  et  al., 
1995).  The  YCSQ  also  provides  differentiated  information 
about  dimensions  of  client  satisfaction,  and  its  directions 
and  item  wording  are  designed  to  minimize  social 
desirability  response  bias. 
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Method 

Participants  were  1 50  clients  and  their  parents  who  had 
completed  therapy  at  a community  mental  health  center  for 
children  in  northeastern  Ohio.  The  age  range  was  1 1-17  and 
the  mean  age  was  13.23  (SD  = 1.72).  The  sample  was  51% 
female.  Ethnic  composition  was  43%  African  American, 
50%  White,  5%  Hispanic,  and  2%  other  ethnic  group.  The 
sample  included  only  subjects  who  had  participated  in  at 
least  three  therapy  sessions  (not  counting  the  initial 
diagnostic  assessment). 

The  primary  income  source  was  employment  for  37%  of  the 
families,  with  63%  receiving  AFDC  or  another  form  of 
income  assistance.  A total  of  62%  of  the  families  were 
headed  by  a single  parent,  and  38%  of  the  families  provided 
two-parent  homes.  Eighteen  percent  of  the  youth  had  a 
history  of  juvenile  court  involvement.  At  intake,  28%  of  the 
sample  were  diagnosed  with  Oppositional-Defiant  Disorder, 
8%  with  Adjustment  Disorder  with  Mixed  Emotional 
Features,  8%  with  Sexual  Disorder  Not  Otherwise  Specified 
(juvenile  sex  offenders),  6%  with  Dysthymic  Disorder,  and 
the  remainder  widely  dispersed  among  a number  of 
diagnoses. 

Most  subjects  participated  in  a combination  of  treatment 
modalities,  so  that  the  following  proportions  add  up  to  more 
than  100%.  Seventy-seven  percent  of  the  subjects  received 
individual  therapy,  91%  family  therapy,  and  22%  group 
therapy. 

Fifty-eight  percent  of  the  consumers  eligible  for  the  study 
became  participants  (N  = 87).  Only  34%  of  the  attrition  was 
due  to  client  unwillingness  to  participate.  The  remainder  of 
non-participants  resulted  from  the  family  not  having  a 
phone,  moves  without  forwarding  information,  the  family's 
inability  to  complete  the  interviews  by  eight  weeks  from 
their  last  appointment,  language  barriers,  and  the  child  no 
longer  being  in  the  home. 

Candidate  items  for  the  YCSQ  were  generated  on  the  basis 
of  a literature  review,  focus  groups  with  clinicians,  and 
individual  exploratory  interviews  with  10  young  clients  and 
their  parents.  This  resulted  in  17  candidate  items.  The 
design  of  the  YSCQ  involved  combining  psychometric 
considerations  with  the  need  to  maximize  ease  and  comfort 
of  responding.  Responses  were  coded  in  the  form  of  a 
four-point  Likert-type  scale.  However,  rather  than 
presenting  the  scale  directly  to  subjects,  responses  were 
obtained  through  very  brief,  scripted  exchanges  between 
interviewer  and  youth,  with  follow-up  questions  when 
necessary.  The  items  were  given  different  scale  anchors, 
depending  on  the  item  content. 

Telephone  administration  of  the  YCSQ  usually  took  about 
10  minutes  unless  there  was  spontaneous  elaboration. 
One-half  of  the  subjects  responded  to  the  questionnaire 
again  in  three  weeks  to  assess  reliability.  One  item  was 
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again  in  three  weeks  to  assess  reliability.  One  item  was 
repeated  as  a reliability  check.  Five  protocols  were  excluded 
because  of  evidence  of  random  responding. 

The  procedure  also  included  the  Child  Behavior  Checklist 
(CBCL;  Achenbach,  1991a)  and  the  Youth  Self-report 
(YSR;  Achenbach,  1991b),  which  were  administered  prior 
to  and  after  therapy  as  an  assessment  of  outcome.  Parents 
also  completed  a measure  of  their  satisfaction  with  services 
(Kotsopoulos,  Elwood,  & Oke,  1989)  during  a telephone 
interview. 

Results 

Because  YCSQ  items  have  different  anchors,  there  were 
questions  about  the  parametric  versus  nonparametric  nature 
of  the  data.  In  order  to  address  this  issue,  we  computed 
Spearman  correlations  for  the  psychometric  analyses 
addressed  below  and  compared  them  to  the  corresponding 
Pearson  correlations.  The  5 1 pairs  of  correlations  were 
extremely  similar,  with  no  significant  differences, 
suggesting  that  YCSQ  scaling  design  does  not  result  in 
marked  departure  from  interval  equality  and  item 
comparability,  so  that  parametric  procedures  are  appropriate 
for  the  data. 

There  were  three  criteria  for  retaining  items  in  the  measure: 
(a)  test-retest  reliability  correlation  ^ .50;  (b)  part-whole 
correlation  ^ .40;  and  (c)  correlation  with  a validation  item 
about  overall  satisfaction  ^ .30  fsee  Table  11.  Fourteen  items 
met  these  criteria.  All  correlations  reported  below  are 
significant  with  p < .0001  unless  otherwise  indicated.  The 
average  reliability  correlation  was  .67,  with  an  average 
part-whole  correlation  of  .65,  and  an  average  validity 
correlation  of  .48.  The  item  with  the  highest  part-whole 
correlation  (.78)  was  the  one  about  overall  satisfaction. 

Total  scores  had  a reliability  correlation  of  .92.  Cronbach's 
alpha  was  .90. 

Total  scores  on  the  instrument  ranged  from  8 to  42,  with  a 
mean  of  30.64  and  a standard  deviation  of  8. 1 1 . The  average 
item  score  was  2.19.  Given  an  item  score  range  of  0 to  3, 
these  results  indicate  that  the  sample  as  a whole  expressed  a 
fairly  high  degree  of  satisfaction  with  their 
services&endash;a  finding  consistent  with  the  general  client 
satisfaction  literature  (Young  et  al.,  1995). 

We  performed  a principal  components  analysis  with 
varimax  rotation  in  order  to  identify  the  major  factors  of 
youth  consumer  satisfaction.  Two  factors  emerged.  The 
Relationship  with  Therapist  factor  included  six  items  asking 
whether  the  youth  liked  the  counselor,  had  a good  time  in 
therapy,  perceived  the  therapist  as  understanding,  caring, 
having  helpful  ideas,  and  as  understanding  his  or  her 
cultural  group.  The  Benefits  of  Therapy  factor  included 
eight  items  asking  about  therapy-caused  change  in  the  areas 
of  feelings,  behavior,  self-concept,  personal  problems,  and 
family  relationships,  and  asking  about  beneficial  learning. 
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family  relationships,  and  asking  about  beneficial  learning, 
understanding  therapeutic  goals,  and  overall  satisfaction. 
Cronbach's  alpha  for  the  individual  factors  was  .85  in  both 
cases.  The  inter-factor  correlation  was  .66. 

In  order  to  assess  convergent  validity,  we  computed 
correlations  between  the  YCSQ  and  several  other  measures 
of  treatment  quality  Csee  Table  2).  YCSQ  scores  were, 
somewhat  positively  related  to  improvement  from  pretest  to 
posttest  on  the  CBCL  (r  = .18,  p < .05),  but  not  the  YSR  (r  = 
-.04,  ns).  YCSQ  scores  had  correlations  of  .37  with 
improvement  in  DSM  III-R  Global  Assessment  of 
Functioning  scores,  .41  with  a therapist  rating  of  treatment 
progress,  .53  with  parent  satisfaction  scores,  and  .47  with  a 
parent  rating  of  treatment  progress. 

Boys  produced  higher  YCSQ  scores  than  girls  (t  = 2.00,  p < 
.05).  There  was  a marginally  significant  correlation  with  age 
(r  = -.15,  p < .07).  African  American  and  White  clients 
produced  similar  scores  (t  = .29).  Client  satisfaction  did  not 
differ  depending  on  parental  employment  status  (t  = 1 .67), 
single  versus  two-parent  families  (t  = .56),  past  treatment 
history  at  our  agency  (t  = .58)  or  another  setting  (t  = .59),  or 
youth  involvement  in  the  juvenile  justice  system  (t  = .52). 
YCSQ  scores  showed  a marginally  nonsignificant 
correlation  of  .16  (p  < .06)  with  number  of  sessions. 

The  interview  included  two  questions  asking  whether  the 
youth  wanted  to  receive  counseling  or  if  they  were  required 
to  come.  There  was  no  relation  between  client  satisfaction 
and  whether  the  youth  chose  or  were  required  to  begin 
counseling  (r  = .12).  Youth  who  reported  voluntary 
participation  in  therapy  by  the  time  of  termination  were 
more  satisfied  with  their  services  (r  = .50). 

Conclusions 

The  results  suggest  that  the  YCSQ  is  a reliable,  internally 
consistent,  and  valid  measure  of  satisfaction  of  mental 
health  services  for  young  clients,  however,  more  research  is 
needed.  The  14  retained  items  demonstrated  satisfactory 
psychometric  properties  and  an  interpretable  factor  structure 
suggesting  that  the  relationship  with  the  therapist  and  the 
perceived  benefits  of  counseling  are  two  main  components 
of  overall  youth  satisfaction.  Youth  client  satisfaction 
showed  significant  relations  with  5 of  6 other  measures  of 
treatment  effectiveness  (with  the  exception  of  the  YSR), 
providing  evidence  of  convergent  validity  for  the  YCSQ. 

Telephone  interviews  proved  to  be  a feasible  method  of 
administration.  Given  that  mailed  questionnaires  often  have 
low  return  rates  and  that  terminations  are  often  unplanned, 
the  telephone  method  seems  to  have  practical  advantages. 
However,  youth  may  not  have  felt  the  survey  was 
anonymous  when  conducted  by  phone  by  staff  from  the 
center,  leading  to  bias  towards  positive  responses 
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The  descriptive  results  suggest  that  most  young  counseling 
clients  express  a fairly  high  level  of  satisfaction  with  their 
services  even  when  directions  and  items  wording  are 
designed  to  give  permission  for  dissatisfied  as  well  as 
satisfied  responses.  Clients  who  began  treatment 
involuntarily  ended  with  the  same  level  of  satisfaction  as 
youth  who  initially  wanted  to  receive  therapy.  Interpretation 
of  these  results  should  be  cognizant  of  the  possibility  that 
clients  who  had  dropped  out  prior  to  their  third  therapy 
session  and  those  who  were  not  recruited  for  the  study  may 
have  had  a lower  level  of  satisfaction  than  our  study 
participants. 
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Introduction 

The  purpose  of  this  study  was  to  examine  the  need  for  and 
use  of  mental  health  services  from  both  adolescents'  and 
service  providers'  perspectives.  This  study  examined  the 
mental  health  needs  of  adolescents  who  interact  with  four 
types  of  service  sectors  that  typically  function  as  gateways 
to  mental  health  services  (i.e.,  education,  child  welfare, 
juvenile  justice,  and  primary  health).  We  compared 
prevalence  estimates  collected  from  the  youths  themselves 
to  counts  of  clients  kept  by  the  gateway  providers  and 
comments  made  in  focus  groups  by  these  providers 
regarding  their  understanding  of  mental  health  service 
issues.  By  comparing  these  three  sets  of  data,  we  obtained  a 
unique  picture  of  the  relationship  between  the  services 
offered  to  youths,  the  youths'  need  for  services,  and  the 
understanding  of  that  need  by  the  providers. 
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Method 


The  Youth  Services  Project,  funded  by  the  National  Institute 
of  Mental  Health,  examined  adolescents'  mental  health 
needs  and  their  use  of  mental  health  services.  The  Youth 
Services  Project  (a)  interviewed  youths  (N  = 796)  from  St. 
Louis  city  juvenile  justice,  education,  primary  health  care, 
and  child  welfare  sectors  (approximately  200  youth  per 
sector);  (b)  gathered  anonymous  counts  of  the  mental  health 
clients  from  each  sector;  and  (c)  conducted  focus  groups 
with  providers  from  each  sector.  All  four  gateway  service 
sectors  were  in  a position  to  screen  youths  for  mental  health 
problems  and  to  either  provide  some  mental  health  services 
or  refer  to  services  elsewhere. 

Data  Sample 

We  recruited  subjects  with  the  aid  of  service  providers,  by 
having  interviewers  approach  youths  in  the  service  sector 
waiting  rooms,  and  by  letters  and  posters  requesting 
volunteers  from  each  sector's  service  users.  The  interview 
participants  were  all  between  14  and  17  years  of  age,  with  a 
mean  age  of  15.3  years.  Thirteen  percent  were  white,  86  % 
Black,  and  1 % of  other  ethnic  background.  Forty-three 
percent  were  male,  and  57  % female.  The  occupation  of  the 
parent  who  provided  the  most  financial  support  to  the 
family  in  the  last  6 months  determined  the  youth's 
socioeconomic  status.  Accordingly,  15%  were  welfare 
recipients,  3 % laborers  or  semiskilled  workers,  23  % blue 
collar,  14  % white  collar,  and  39%  professional. 

Data  Sources 

The  first  source  of  data  came  from  highly  structured 
interview  protocols  which  yielded  information  concerning: 
(a)  demographics;  (b)  social  and  academic  behaviors;  (c) 
mental  health;  (d)  service  use;  and  (e)  family,  social,  and 
community  environments.  Measures  of  mental  health 
problems  came  from  the  Diagnostic  Interview  Schedule  for 
Children-Revised  (DISC-R:  Schaffer,  Schwab-Stone, 

Fisher,  Cohen  et  al.,  1993). 

The  second  source  of  data  was  derived  from  service  sector 
tallies  of  those  adolescents  were  sector  clients  during  the 
same  period  in  which  we  recruited  and  interviewed  our 
service  sample.  These  were  anonymous  counts  collected  by 
the  service  providers,  detailing  the  demographics  of 
adolescent  health  service  users  and  their  mental  health 
problems.  For  a one  month  period  during  the  six  month 
interview  field  work,  service  providers  at  each  sector 
(except  primary  health)  completed  an  anonymous  tally  sheet 
indicating  the  age,  gender,  race,  and  type  of  mental  health 
problem  of  each  consecutive  youthful  client.  The  health  care 
sector  already  had  such  information  as  part  of  their  billing 
records,  they  provided  to  us  for  that  one  month  period. 
Pre-printed  tally  sheets  were  distributed  to  sector  providers 
in  order  to  maximize  the  ease  and  accuracy  of  tally-keeping 
by  the  service  providers. 
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by  the  service  providers. 

The  third  and  final  source  of  data  came  from  focus  groups 
held  with  service  providers  from  each  sector.  Their 
professions  varied  in  accord  with  the  sector:  medical  doctors 
and  nurses  from  the  Health  Sector,  social  workers  from 
Child  Welfare,  deputy  juvenile  officers  from  Juvenile 
Justice,  and  counselors  and  social  workers  from  Education. 

Results 

Youths'  Need  for  Services 

The  percentages  of  youths  meeting  criteria  for  a diagnosis  of 
depression  ranged  from  8%  to  17%  depending  on  the  sector 
providing  services;  for  conduct  disorder,  the  range  was  8% 
to  30%  ; and  finally,  for  substance  dependence/abuse,  the 
range  was  5%  to  27%.  The  range  for  suicidality  was  7%  to 
17%. 

Services  for  Mental  Health  Problems 

Our  interviews  indicated  that,  in  general,  half  of  the  youths 
with  mental  health  problems  reported  being  helped  by  the 
gateway  sector  where  we  recruited  them  (see  Table  1~). 
However,  the  rates  varied  by  sectors.  The  health  sector 
interviewees  reported  the  lowest  service  rate  among  all 
sectors,  with  less  than  one  fifth  of  youths  with  problems 
receiving  services.  In  the  child  welfare  sector,  the 
percentages  of  youths  who  had  problems  and  who  received 
help  ranged  from  54%  to  64%  depending  on  the  mental 
health  problem.  In  the  education  sector,  the  percentages  of 
interviewees  reporting  services  ranged  from  50  to  80%.  The 
highest  percentages  of  mental  health  services  were  reported 
by  interviewees  from  the  juvenile  justice  sector,  with  a 
range  from  80  to  95%. 

Comparison  of  Interviews  and  Anonymous  Counts 

Table  1 allows  the  reader  to  compare  percentages  for  the 
following:  (a)  the  percent  of  interviewees  identified  as 
meeting  criteria  for  diagnoses  of  a mental  health  problem; 

(b)  the  percent  of  interviewees  who  were  having  significant 
symptoms  of  such  a problem;  and  (c)  the  percent  of  sector 
clients  identified  through  the  anonymous  counts  as  having 
such  problems. 

In  the  child  welfare  sector,  the  counts  (when  compared  to 
the  interview  data)  underidentified  youths  with  drug  or 
alcohol  abuse  problems 

(1%  versus  17%).  In  contrast,  the  child  welfare  sector 
counts  overidentified  youths  as  having  significant  conduct 
disorder  problems  (46%  versus  1 7%),  but  approximated  the 
same  rates  as  the  interviews  for  suicidal  youths  (18%  versus 
17%)  and  depression  (19%  versus  17%).  Within  the  juvenile 
justice  sector,  the  anonymous  counts  identified  similar 
percentages  of  mental  health  problems  to  those  found  in  the 
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percentages  of  mental  health  problems  to  those  found  in  the 
youth  interviews.  The  education  sector  tally  underidentified 
clients  as  having  problems  with  drugs  or  alcohol  (0.1% 
versus  5%),  being  suicidal  ( 0.7%  versus  8%),  and  being 
depressed  (3%  versus  8%  ),  but  overidentified  behavior 
problems  (14%  versus  8%).  For  the  227  consecutive 
adolescent  patients  tallied  by  providers  from  the  health  care 
sector,  none  were  identified  as  having  any  mental  health 
problems. 

Focus  group  comments 

Many  of  the  comments  offered  by  the  providers  during  the 
focus  groups/interviews  gave  significant  clues  as  to  why 
their  particular  sector  did  or  did  not  identify  or  serve  youths 
who  had  mental  health  problems.  The  juvenile  justice 
sector,  which  had  the  highest  rates  of  identification  of 
problems,  also  had  a special  diagnostic  and  screening  unit 
attached  to  it.  Although  the  workers  participating  in  the 
focus  group  reported  that  providing  mental  health  services 
was  outside  their  scope  of  activities,  they  were  aware  of  the 
necessity  for  mental  health  assessment  and  even  commented 
that  watching  office  interactions  was  particularly  helpful  to 
them. 

The  child  welfare  sector  had  the  next  highest  rate  for 
identification  of  problems.  Those  workers  also  commented 
that  providing  mental  health  services  was  outside  their 
scope  of  action.  Nevertheless,  they  felt  that  they  should 
identify  such  problems.  They  pointed  out  that  car  rides  with 
the  teenagers  promoted  freer  discussion  and  were  thus 
particularly  helpful  in  assessing  the  youths'  mental  health 
problems. 

The  educational  sector  had  particularly  low  identification 
rates  for  drug  and  alcohol  abuse  and  suicidality.  The 
providers  reported  that  it  was  hard  for  them  to  identify 
mental  health  problems  because  parents  and  the  community 
often  denied  that  the  problems  existed.  They  also  identified 
lack  of  time  and  very  heavy  caseloads  as  barriers  to 
identifying  youth  problems.  They  also  reported  that  the 
absence  of  support  groups  for  adolescents  with  mental 
health  problems  kept  them  from  identifying  and  referring 
for  such  problems. 

The  health  sector  identified  no  youths  as  having  mental 
health  problems.  Of  all  the  sector  focus  groups,  health  care 
workers,  when  asked  about  the  kinds  of  problems  their 
adolescent  clients  had,  did  not  list  one  behavioral  or  mental 
health  problem.  They  also  mentioned  that  they  lacked 
competence  in  assessing  and  treating  mental  health 
problems,  and  were  uninformed  about  available  resources 
for  referral.  When  they  referred  youths  for  care,  they  felt 
that  the  long  waiting  periods  for  mental  health  appointments 
(up  to  one  month)  was  a significant  barrier.  The  health  care 
workers  also  mentioned  their  concern  with  lack  of 
continuity  care. 
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Conclusions 

The  interviews  showed  that  significant  numbers  of  youths  at 
each  of  the  four  sectors  had  mental  health  problems. 
However,  only  a limited  percentage  of  those  youths  with 
mental  health  problems  received  services  from  their 
gateway  sector.  This  percentage  varied  widely  by  sector, 
with  the  juvenile  justice,  education,  and  child  welfare 
sectors  providing  services  to  a much  higher  percentage  of 
youths  than  the  health  sector. 

The  anonymous  counts  paralleled  the  interview  data.  The 
juvenile  justice  sector  tally  reported  the  highest  number  of 
clients  with  problems.  The  child  welfare  sector  tally 
identified  a somewhat  lower  rate  as  having  problems, 
despite  the  interview  data  showing  an  excellent  record  for 
services.  The  education  sector  had  a very  large  discrepancy 
between  the  youths'  interview  data  and  the  providers'  tally 
data,  although  many  of  the  youths  with  mental  health 
problems  felt  they  were  receiving  help.  The  health  sector 
had  the  worst  service  record  from  both  interview  data  and 
providers'  tally  data. 

This  study  focuses  attention  on  several  important  issues  for 
service  delivery.  Many  of  the  adolescents  report  that  their 
gateway  providers  (e.g.,  school  counselors,  physicians,  child 
welfare  workers,  or  deputy  juvenile  officers)  neither 
provided  services  nor  informed  them  that  care  might  be 
available.  Further,  many  of  the  providers'  records  indicate 
unawareness  of  the  extent  of  need.  This  would  indicate  that 
the  gateway  providers'  perception  of  need  (although  often 
inaccurate)  may  determine  the  youth's  pathway  to  services 
more  than  actual  need  and  service  availability.  We  must 
therefore  educate  gateway  providers  in  diagnostic  and 
assessment  procedures,  kiiown  incidences  for  mental  health 
problems,  service  skills,  service  availability,  and  referral 
procedures. 
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